Selection of individuals from a pool of candidates in a competition system

ABSTRACT

An integrated highly automated competition system allows for selection of one or more competition winners. Documents can be submitted electronically or by hardcopy. Document submissions are checked against databases to determine duplicate existence by a variety of information extracted from the documents. Candidates are filtered by one or more filters requiring candidates meet completeness, qualification, eligibility, and partner/discretionary standards. Candidates are evaluated by one or more readers and scored on a plurality of cognitive and non-cognitive variables. Readers are monitored and corrective action is taken when reader evaluations are determined to require corrective action. Candidates are selected by a two stage process first using raw read scores and then using a successive weighted read score iteration or tie-breaking stage. Winning candidates are subject to progress tracking and re-qualification prior to receiving award payments.

FIELD OF THE INVENTION

[0001] The present invention relates to the evaluation of humanresources, and more particularly to the automation-enhanced selection ofindividuals from a pool of competition candidates based on evaluationsof cognitive and non-cognitive variables relative to a competitionstandard.

BACKGROUND OF THE INVENTION

[0002] The appropriate selection of individuals or candidates foremployment, appointment to political positions, promotions, receivingawards or scholarships, and so on has traditionally been a difficultproblem.

[0003] One trend facing competition systems is that of ever-increasingnumbers of applications being received by competition systems. Thisresults in problems of data handling, of selecting the best candidatesfrom an ever-increasing number of highly-qualified applicants, and alsoof eliminating an ever-increasing number of lower-quality, ineligible,or incompletely-responsive candidates.

[0004] The problem of candidate selection is compounded by candidatefraud which is becoming increasingly sophisticated. The forging ofdocuments, the use of documents or credentials of others having similarnames or demographic information, and so on complicates competitionsystems due to the detection and resolution requirements which must beimposed, as well as the resultant unfairness when such fraud goesundetected.

[0005] Another problem is the occurrence of duplicate forms. This oftenoccurs as individuals apply multiple times in order to correctinformation or perhaps even in the belief that this can aid theirchances of success.

[0006] The detection of both duplicates and fraud is furthercomplicated, however, in that situations arise where two or morelegitimate candidates sometimes appear identical from their submittedpapers. Such situations often arise in connection with twins havingsimilar or identical names, for example.

[0007] The selection of candidates for positions of futureresponsibility such as political appointees, military or civilianofficers or leaders, corporate management, academic scholars, and so onhas at its root the problem of selecting the best candidate based onpredicted future performance. Studies have shown that non-traditionalvariables such as non-cognitive variables are sometimes a betterpredictor of future performance than traditionally-used variables.

[0008] The use of these variables, as well as the use of traditional butsubjective variables, such as essay tests, has a problem because thenature of their evaluation is subjective and thus difficult, even for asingle evaluator, to do in a fair and consistent manner. This problemincreases with the number of evaluators and encompasses the problems ofevaluator selection, training, and monitoring. Studies have shown, forexample, that evaluators sometimes vary their decision methodologies intheir evaluations, especially just before breaks or at the end of theday. Furthermore, studies have also shown that evaluators may evaluatenon-cognitive variables more harshly for candidates of one ethnic orcultural background as compared with candidates of other ethnic orcultural backgrounds. The monitoring of evaluators is additionallycomplicated as system generally don't have the infrastructure necessaryto allow real-time monitoring in a non-invasive way or the ability toimplement corrective measures in real-time.

[0009] Additionally, traditional systems are lacking in methodologies totrack both competition winners and competition losers for later use inproducing statistical support for and improvement of system selectioncriteria.

[0010] From the above, it is evident that improvements are needed in theareas of data management (i.e. duplicate identification, false-duplicateidentification, and fraud identification), candidate pool reduction,evaluator management, candidate selection, and winning-candidateprogress tracking.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011]FIG. 1 shows a block diagram of an embodiment of a system for theselection of individuals from a pool of candidates;

[0012]FIG. 1a shows a block diagram of an embodiment for database 104 ofFIG. 1;

[0013]FIG. 2 shows a block diagram of an embodiment for a high-levelsoftware architecture for the system of FIG. 1;

[0014]FIGS. 3a-3 b show a block diagram in greater detail of thesoftware architecture of FIG. 2;

[0015]FIGS. 3c-3 d show the block diagram of FIGS. 3a-3 b emphasizingthe interconnections of the reporting module (280).

[0016]FIG. 4 shows a general flowchart of steps traversed by anexemplary method in operation of the systems of FIGS. 1-2;

[0017]FIG. 5 shows a flowchart of steps traversed by an exemplary methodin the operation of electronic reception of documents in the systems ofFIGS. 1-2;

[0018]FIG. 6 shows a flowchart of steps traversed by an exemplary methodin the operation of hardcopy reception in the systems of FIGS. 1-2;

[0019]FIG. 7 shows a flowchart of the steps traversed in an exemplaryalgorithm for the storing and duplicate detection for the submissionprocedures shown in FIGS. 5-6;

[0020]FIG. 7a shows a flowchart of an exemplary embodiment for possibleduplicate document review;

[0021]FIG. 8 shows a flowchart of steps traversed in an exemplaryalgorithm the grouping and level 1 filtering for candidate packagecompleteness shown in FIG. 4;

[0022]FIG. 9 shows a flowchart of steps traversed in an exemplaryalgorithm in the operation of the level 2-3 filtering of electronicdocuments shown in FIG. 4;

[0023]FIG. 10 shows a flowchart of steps traversed in an exemplarymethod in operation of the reading process of FIG. 4;

[0024]FIG. 11 shows a flowchart of steps traversed in an exemplarymethod in operation of the selection and training of readers toparticipate in the process of FIG. 8;

[0025]FIGS. 12a-12 b show a flowchart of steps traversed in an exemplaryalgorithm in operation of finalist selection of FIG. 4;

[0026]FIGS. 12c-12 e shows an example of the steps traversed in anapplication of the tie-breaking procedure of FIG. 12b.

[0027]FIG. 13 shows a flowchart of steps traversed in an exemplarymethod in operation of competition winner confirmation of FIG. 4;

[0028]FIG. 14 shows a flowchart of steps traversed in an exemplarymethod in operation of competition winner tracking of FIG. 4;

[0029]FIG. 14a shows a flowchart of the steps traversed in an exemplarymethod for the tracking of competition winners and non-selectedapplicants;

[0030]FIG. 15 shows an exemplary display presented to an evaluator;

[0031] FIGS. 16-17 show exemplary displays for use during determinationof evaluator eligibility;

[0032] FIGS. 18-19 show exemplary displays for use during determinationof candidate eligibility;

[0033] FIGS. 20-21 show exemplary displays for use during evaluation ofcandidate packages;

[0034] FIGS. 22-25 show exemplary displays of candidate progress detailreports;

[0035] FIGS. 26-28 show exemplary displays of completed applicantpackage reports;

[0036] FIGS. 29-30 show exemplary displays of possible duplicatedocument reports;

[0037] FIGS. 31-34 show exemplary display of candidate detail listingreports;

[0038]FIG. 35 shows an exemplary display of a nominator floater withpossible key form report;

[0039] FIGS. 36-37 show exemplary display of nominator floater detailreports;

[0040] FIGS. 38-40 show exemplary display of nominator detail reports;

[0041]FIG. 41 shows an exemplary display of a recommender floater withpossible key form report;

[0042] FIGS. 42-43 show exemplary display of recommender floater detailreports; and

[0043] FIGS. 44-46 show exemplary display of recommender detail reports.

DETAILED DESCRIPTION

[0044] Shown and described are embodiments, automated andpartially-automated, of a system and method for selecting one or morecandidates from a pool of candidates. It is noted that while thisdisclosure illustrates the present invention by examples drawn mainlyfrom academic scholarship competitions, the present invention hasapplication in many fields outside of academics such as, but not limitedto: employee hiring; employee to job, position, or task matching;military officer selection; political appointee selection; personal dateselection; personal friend selection; and so forth.

[0045] Referring to FIG. 1, shown is a block diagram of an embodimentfor a competition system (100) for the selection of individuals from apool of candidates.

[0046] Shown are system processing hardware (102), database hardware(104), a non-select segment (112), a candidate segment (114), a scholarsegment (116), a client device (120), a network (122), network interfacehardware (124), an envelope (130) representing postal delivery, ascanning and processing system (132), a workstation (140), a workstation(142) labeled “workstation n”, a correspondence printer (150), and acheck printer (152).

[0047] The system processing hardware (102) is coupled to the databasehardware (104). The database hardware (104) as shown comprises thenon-select segment (112), the candidate segment (114), and the scholarsegment (116). The client device (120) is coupled to the network (122).The network (122) is coupled to the network interface hardware (124),which, in turn, is coupled to the system processing hardware (102). Theenvelope (130) representing postal delivery is coupled to the scanningsystem (132) which, in turn, is coupled to the system processinghardware (102). The workstation (140) and the workstation (142) labeled“workstation n” are both coupled to the system processing hardware(102). The correspondence printer (150) and the check printer (152) arealso both coupled to the system processing hardware (102).

[0048] The system processing hardware (102) carries out processingrequired for the competition system (100). The system processinghardware (102) can be implemented by any hardware suitable for runninggeneral purpose software such as, but not limited to, any commercialcomputing hardware. By way of example, most computing systems having oneor more Intel-, AMD-, or Motorola-based processors and targeted forgeneral home or business use can be used for the system processinghardware (102).

[0049] The database hardware (104) stores information relating tocandidates required in carrying out the processing of competitions. Theexact nature of the storage architecture is not dependant on the presentinvention and can take any form necessary to the particularimplementation. In one embodiment, as shown, the database hardware (104)is segmented to manage individuals during various stages ofconsideration in the competition system (100). In this embodiment, datais marked to segment or group the data. The data is segmented or groupedinto the categories which include applicants (the applicant category orsegment (114)), applicants who are not selected (the non-select categoryor segment (112)), and candidates selected as competition winners (thecompetition-winner category or segment (116) also called the scholarcategory or segment in one embodiment). In one variation, the databasecan be any suitable commercial database product such as, but not limitedto, products by Oracle, Sybase, or Microsoft.

[0050] In another embodiment, the database hardware (104) is physicallydifferentiated into two or more separate databases as required. In oneembodiment, the database hardware (104) includes a staging or applicantdatabase, a non-select database, a candidate or nominee database (forapplicants who have passed qualification and eligibility processes), anda competition-winner or scholar database. This embodiment is used, forexample, where existing hardware capacities mandate such an approach orin systems which have an absolute requirement of data separation such asfor security reasons. In systems requiring higher separation of data, afifth database for the storage of finalists prior to confirmation, afinalist database, can be added.

[0051] In an embodiment, applications, to be complete, comprise two ormore documents. In a further variation, an applicant package (alsocalled a candidate package or profile) includes an applicant form(submitted by the applicant), also called a nominee or candidate form, anominator form (submitted by an individual with academic knowledge ofthe applicant such as a school teacher or administrator), and arecommender form (submitted by an individual with personal knowledge ofthe applicant such as a family friend. It is noted that school officialsand teachers can also be nominators). In this embodiment, the applicantsegment (114) comprises applicant-related submissions received from boththe electronic and hardcopy submissions. The non-select segment (112)comprises applicant packages for applicants who ultimately do not win acompetition, i.e., for applicants who are ultimately not selected or forapplicants having incomplete or ineligible applicant packages. Thecompetition-winner segment (116) (also called the scholar segment inacademic scholarship embodiments) comprises candidate packages relatingto those candidates who ultimately win a competition.

[0052] When application documents are received by the system, they arestored into the database (104), associated with any other records (thedatabase versions of submitted documents) relating to the sameapplicant, and marked as being in the applicant segment (114). Later,the associated records (i.e. the applicant packages) are submitted toqualification and eligibility procedures. Applicant packages do not passeither the qualification and eligibility procedures are marked as beingin the non-select segment (112). Later still, the applicant packageswhich have not been non-selected are evaluated, scored, and one or morecompetition winners selected. In one embodiment, once an applicant isselected as a competition finalist, he or she is referred to as acandidate and their applicant package is referred to as a candidatepackage. The candidate packages for those candidates who ultimately wina competition are marked as being in the competition-winner segment(116) upon certification (verification or confirmation of data suppliedin the applicant, nominator, and recommender forms) and acceptance ofcompetition winning status (i.e. acceptance of the competition award) bythe candidate. Further to this embodiment, candidates who decline thecompetition award, or who are not certified, are designated asnon-selects and their candidate packages are marked as being in thenon-select segment (112).

[0053] In one embodiment, in order to keep track of the status ofindividuals during the competition process, various lists are maintainedor the ability to create them at any time is preserved. These listsinclude an applicant list for listing all applicants for which anydocument or form has been submitted, a non-select list for listing allindividuals who ultimately are not winners of the competition, acandidate list for all applicants passing an initial qualification andeligibility process, a finalist list for listing all candidates selectedas finalists, and a competition-winner list for listing all finalistswho pass a confirmation and acceptance process. Further to thisembodiment, these lists are protected and each require (possiblydifferent) special system privileges to have access. In a variation ofthis embodiment, each list can include various data regarding theindividuals listed. The non-select list can include, for example, thedate and stage within the competition at which the individual wasnon-selected or the competition-winner list can include dates at whichthe competition winner is expected to submit re-qualification documents.

[0054] Information relating to applicants can be submitted by anyconventional method such as by postal mail or electronic submission.Applicant information submitted by postal mail generally takes the formof a paper document (including predefined forms after user completion ornonstandard documents such as letters), but other manifestations, suchas, but not limited to, computer readable media (containingwordprocessing files or even image scans of paper documents) arepossible. As used herein, the term “document” refers to any submissionsof applicant information and includes, for example, paper submissions,scanned images on computer-readable media, or any other media or vehiclecontaining information to be incorporated into the competition system.

[0055] Electronic submission begins when a submitted document istransmitted by the client device (120) via the network (122) to thecompetition system (100) where it is received via the network interfacehardware (124). The client device (120) can be any user device able toinitiate document submissions. By way of example, a client device (120)can be, but is not limited to, a computer, a handheld device such as apersonal digital assistant (PDA), or a computer network such as abusiness, college, or university intranet. The network (122) can be anyelectronic or computer network or interconnection of networks such asone or more intranets or internets. In one embodiment, the network (122)is the Internet. The network interface hardware (124) is the physicalconnection of the competition system (100) to the network (122). Thenetwork interface hardware (124) can be any device such as, but notlimited to, a modem, a T1 interface, and so forth. Electronicsubmissions are discussed in greater detail later herein in reference toFIG. 5.

[0056] Documents submitted by postal mail are scanned in at the scanningsystem (132) and thereby incorporated into the system. In the case ofsubmissions on computer-readable media, the candidate information isread in at a computer workstation such as the workstation (140) or(142). Hardcopy submissions are discussed in greater detail later hereinin reference to FIG. 6.

[0057] In one embodiment, the overall selection process has severalstages including an application stage, a qualification stage, anevaluation stage, a ranking and selection stage, a finalist stage, aconfirmation stage, and a tracking or monitoring stage. The applicationstage is where individual applications are received by the competitionsystem (100) as discussed previously. The qualification stage implementsbasic qualification and eligibility requirements and is discussed ingreater detail later herein in reference to FIGS. 8 and 9. Theevaluation stage implements evaluation of qualified applicants and isdiscussed in greater detail later herein in reference to FIG. 10. Theranking and selection stage implements the ranking of evaluatedcandidates and the selection of finalists. Finalists are candidates whoare selected after the evaluation process, but who must pass acertification process before becoming unqualified competition winners.The ranking and selection process is discussed in greater detail laterherein in reference to FIGS. 12a-12 b.

[0058] Once finalists have been identified, the confirmation stageimplements confirmation of the qualifications of finalists and allowsfor ultimate acceptance by confirmed finalists. The confirmation offinalists is discussed in greater detail later herein in reference toFIG. 13. The tracking or monitoring stage implements the tracking ofboth competition winners (i.e., monitoring the progress of finalists whohave accepted their competition awards) and non-selected candidates (forcomparison purposes), Competition award payment and correspondence, suchas notice letters and requests for progress reports, are generated bythe payment and correspondence generation system during the tracking andmonitoring stage. The tracking of competition winners is discussed ingreater detail later herein in reference to FIG. 14.

[0059] The workstation (140) facilitates user use of the competitionsystem (100). The competition system can be used though a singleworkstation but can also be implemented with any number of workstationsnecessary as indicated by the workstation (142) labeled “workstation n”.The workstation (140) can be any computing device able to allow displayand entry of information such as, but not limited to, a personalcomputer, a handheld computing device, a laptop computer, a notebookcomputer, a SPARC (tm) station, etc. The workstation (140) has at leasta display, such as a text or graphical monitor to display informationand one or more input devices such as a keyboard and a mouse to allowentry and manipulation of information.

[0060] The correspondence printer (150) prints correspondence and thecheck printer (152) prints checks. While two printers are shown, anysuitable number of printers can be used such as a report printer for useby the reporting module (280) discussed in greater detail later hereinin reference to FIG. 2. Alternatively, printing can be partially ortotally outsourced.

[0061] Referring to FIG. 1a, shown is a block diagram of an embodimentfor database (104) of FIG. 1.

[0062] Shown are the database (104), an applicant database (162), ascholar database (164), an analytical database (166), an assessmentdatabase (168), a research database (170), and an archive database(172).

[0063] In this embodiment, the applicant database (162) and the scholardatabase (164) are equivalent, respectively, to the applicant segment(114) and the competition-winner segment (116) of FIG. 1. The analyticaldatabase (166) stores information relating to analytical studies ofcandidates in the system to determine trends. The assessment database(168) stores information relating to assessments and other input fromvendor, donor, and other entities regarding the competition system(100). The research database (170) stores information relating toresearch regarding improvement of the rubric or application thereof usedby the competition system (100). Generally, the results stored in theanalytical database (166) is developed from the data stored in theresearch database (170). The archive database (172) stores any datawhich needs to be archived.

[0064] Referring to FIG. 2, shown is a block diagram of an embodiment ofa high-level software architecture for the system of FIG. 1.

[0065] Shown are a document reception subsystem (210), a databaseinterfacing subsystem (220), a qualification and eligibility subsystem(230), an evaluation and selection subsystem (240), a confirmationsubsystem (250), a tracking and re-qualification subsystem (260), acorrespondence and check printing interface subsystem (270), a reportingmodule (280), an external interface module (290), and a financialsubsystem (295).

[0066] The document reception subsystem (210) is coupled to the databaseinterfacing subsystem (220), the confirmation subsystem (250), and thetracking and re-qualification subsystem (260). The qualification andeligibility subsystem (230) is coupled to both the evaluation andselection subsystem (240) and the database interfacing subsystem (220).The evaluation and selection subsystem (240) is also coupled to theconfirmation subsystem (250) and the database interfacing subsystem(220). The confirmation subsystem (250) is additionally coupled to thedatabase interfacing subsystem (220), the tracking and re-qualificationsubsystem (260), and the financial subsystem (295). The tracking andre-qualification subsystem (260) is also additionally coupled to thedatabase interfacing subsystem (220). The financial subsystem (295) iscoupled to the confirmation subsystem (250), the tracking andre-qualification subsystem (260), and the correspondence and checkprinting interface subsystem (270). The reporting module (280) and theexternal interface module (290) are coupled to each other and both arecoupled to all the other subsystems.

[0067] The document reception subsystem (210) handles the incorporationof incoming documents into the competition system (100). While documentscan be received in a variety of ways, such as electronically or by mail,the document reception subsystem (210) incorporates the documents intothe system in a flat file format which is an electronic file containingthe data of the original document. Flat files contain their data in apredefined format understandable by the various subsystems of thecompetition system (100). In one embodiment, the flat file format is acomma delimited format such as a comma separated value (CSV) format.This stored version of a document is referred to as a record, databaserecord, or document record.

[0068] The qualification and eligibility subsystem (230) implementsbasic qualification and eligibility requirements. The qualification andeligibility subsystem (230), in one embodiment, has one or more filters.A filter applies one or more criteria to an applicant package anddetermines whether the applicant package meets or does not meet thecriteria. In one embodiment, applicant packages which do not meet anycriteria are disqualified and marked so as to be in the non-selectsegment (112). In other embodiments, some criteria can be discretionaryand applicants are given the chance to correct deficiencies. In oneembodiment, the filters initially determine applicant packagecompleteness (both data completeness and document record completeness).In this embodiment, once an applicant package is determined to havecompleteness, the applicant package is marked to be in the candidatesegment (114) and is now called a candidate package. After this, furtherfilters apply basic eligibility and discretionary eligibilityrequirements on the candidate packages.

[0069] The evaluation and selection subsystem (240) implementsevaluation and scoring of candidate packages. The evaluation andselection subsystem (240) further compare scored candidate packagesagainst each other and determines competition finalists. Competitionfinalists are those candidates which have been selected as competitionwinners, but are still required to pass a confirmation and acceptanceprocedure. In another embodiment, the confirmation and acceptanceprocedures are dispensed with, and the candidates selected by theevaluation and selection subsystem (240) directly become competitionwinners. This embodiment is used in competitions which do not placefurther responsibility or requirements on competition winners. By way ofexample, awards of recognition, such as a literary award, can be givenwithout requiring confirmation or acceptance.

[0070] The confirmation subsystem (250) implements procedures forconfirming finalists as competition winners and also implements theprinting of correspondence alerting a finalist of his or her status andnotifying them of the requirements and deadlines for submittingconfirmation documents. In one embodiment, if a finalist passes theconfirmation and acceptance procedures, the finalist becomes acompetition winner and the candidate package of the finalist is markedas being in the confirmation-winner segment.

[0071] The tracking and re-qualification subsystem (260) implements theprocedures for tracking competition winners which includes the printingof correspondence notifying and alerting a competition winner todeadlines and requirements for staying qualified and the actualre-qualification determination for competition winners. Re-qualificationis imposed on competition winners for whom a continuing performancecriteria is required. By way of example, an academic scholarship mayrequire a continuing minimum level of performance as measured by gradepoint average (GPA) or a minimum level of course difficulty.

[0072] The correspondence and check printing interface subsystem (270)interfaces between other subsystems and the printing hardware forprinting correspondence, such as on the correspondence printer (150),and checks, such as on the check printer (152).

[0073] The database interfacing subsystem (220) allows the othersubsystems access to the database hardware (104) of the competitionsystem (100).

[0074] The reporting module (280) generates any reports documentingprocesses or results which take place. These reports include, but arenot limited to, error reports such as scanning error or recognitionerror reports, status reports such as applicant status reports, auditreports (e.g. audit trails), possible duplicate reports, complete andincomplete candidate package reports, measurement reports, candidatedetail reports, unmatched document (i.e. floater) reports, progressreports, documentation reports, and donor or other external reports.Documentation reports can include reports which document when and whatprocedures are used during competitions. Such documents can be valuablein a legal sense for sensitive competitions in which the losingcandidates may attempt to make allegations of impropriety. Donor andother external reports can be produced ad hoc to provide specific anddetailed information to the recipients. Donors, for example, may wish toknow how their donations are used and reports addressing this mayinclude amounts paid out, when, and how apportioned.

[0075] The external system interface module (290) enables the system tointerface to other external application systems such as, but not limitedto, financial systems, human resources (HR) systems, and registrationsystems.

[0076] The financial subsystem (295) implements the financial andaccounting aspects of the system including keeping track of donations,award payments, and system costs. In an embodiment for the payment ofacademic scholarships, the financial subsystem (295) manages the awardpayments to scholars. In this embodiment, once an applicant is confirmedas a finalist, the financial subsystem (295) begins keeping track of thescholar. The financial subsystem (295) does this in yearly installmentsby reviewing the scholars package (i.e. the set of documents retained bythe system on the scholar) and calculates a current term award bysumming the monetary amounts of all scholarships (including 3^(rd) partyscholarships), grants, and so forth and subtracts this amount from thetotal tuition amount. The resulting amount is divided by the number ofsemesters, trimesters, etc. applicable to arrive at a base award amount,also called the current term award. Note that in this embodiment, moneyfrom programs such as loans, work study, etc. are not included in thescholarship and grant calculation as the goal of this embodiment is toallow the student to maximize study time without incurring debt forlater repayment. After arriving at the current term award, the financialsubsystem (295) calculates an adjustment to the current term award whichtakes into account the effects of short-term scholarships, differencesin tuition due to special circumstances such as study abroad or changinginstitutions, etc. to arrive at a corrected current term award. Thisamount is then certified and results in payment generated and given tothe scholar.

[0077] The financial subsystem (295) updates the previous calculationsduring the academic year at each term by reviewing the revised scholaraward package and calculating a supplemental award based on the reviewof the revised scholar award package. The revised scholar award packageis generated by adding term-specific information to the scholar awardpackage. It is noted that the financial subsystem (295) accommodatesscholarship payments for summer terms, study abroad.

[0078] Referring to FIGS. 3a-3 b, shown is a block diagram showingdetails of the subsystems of the software architecture of FIG. 2.

[0079] Shown in FIG. 3a are a document reception subsystem (210), aqualification subsystem (230), a selection of competition winnerssubsystem (240), and a database interfacing subsystem (220). Each ofthese subsystems comprise various modules that are discussedindividually as follows.

[0080] The document reception subsystem (210) is shown having a networkinterface module (211), a manual input module (212) a scanner interfacemodule (213), a field extraction and OCR module (214), and an imagestorage (215). OCR is an acronym for optical character recognition.

[0081] The scanner interface module (213) is coupled to the fieldextraction and OCR module (214). The field extraction and OCR module(214) is coupled to the image storage (215). The network interfacemodule (211), manual input module (212), and the field extraction andOCR module (214) are all coupled to the duplicate check/databaseinterface module (221). The image storage (215) is coupled to both theconfirmation module (252) in FIG. 3b (via the connector “A”), and there-qualification module (263) in FIG. 3b (via the connector “B”).

[0082] The document reception subsystem (210) handles the incorporationof incoming documents into the competition system. In one embodiment, asshown, documents can be incorporated by any of three methods: electronicsubmission, hardcopy submission, and manual entry. Electronicsubmissions are received through the network interface module (211) andcan originate from any electronic source such as an internet (e.g. theInternet) or an intranet. Electronic submissions can also be acceptedfrom computers coupled directly to the competition system (and locallytransferring documents in flat file format received on magnetic disc orcompact disc, for example) such as the workstation (140) or theworkstation (142). In one embodiment, the process of electronicsubmission directly produces the flat file format of the inputted formas the user (i.e. the applicant or, in one embodiment, the recommenderor nominator) interacts with the competition system (100) in enteringdata. Electronic data entry by users is discussed in greater detaillater herein in reference to FIG. 5.

[0083] From the network interface module (211), flat files aretransferred to the duplicate check/database interface module (221).Incorporation of flat files is discussed in greater detail later hereinin reference to FIG. 7.

[0084] Documents can also be submitted in paper or other hardcopy formincluding bit-image data on computer readable media such as on magneticdisc or optical disc, for example. Bit-image data refers to files not inflat format, such as bit-mapped images. Media containing bit-mappedimage data generally results from external scanning of paper documentsand storage onto computer-readable media, most likely by the candidatehimself or herself. In the embodiment as shown, the process of hardcopysubmission is received through the scanner interface module (213).Paper, or other hardcopy documents requiring scanning, are first scannedin at one or more scanner modules and incorporated onto the system via ascanner interface module (213). Pre-scanned forms submitted oncomputer-readable media are simply read into the system. Incorporationof documents via scanning is discussed in greater detail later herein inreference to FIG. 6.

[0085] Documents submitted in a non-standard form format or whichotherwise fail the attempt by the field extraction and OCR module (214)to extract field information can be incorporated into the system viarekeying at the manual input module (212). This directly produces thecorrect flat file format and this is then transferred to the duplicatecheck/database interface module (221) where the system storage is firstchecked to determine if the received flat file is already stored in thedatabase hardware (104).

[0086] The scanned forms of the documents are then are transferred tothe field extraction and OCR module (214) where the information of eachfield is extracted and the flat file form of the document is created.Field extraction involves the action of extracting information frompredetermined fields and creating a “flat file” for the scanneddocument. The field extraction process requires that the documents besubmitted using a predefined form (a form having defined “fields” whichare located at predefined locations and are predetermined to holdspecified information such as for a last name, first name, etc.) whichis available upon request or from the Internet website. Use of thepredefined form is encouraged for ease of data incorporation and toensure that all required information is presented in sufficient detail.Once the flat file of a document is created by the field extraction andOCR module (214), it is transferred to the duplicate check/databaseinterface module (221).

[0087] If the flat file already exists as a record in the databasehardware (104), the system does not immediately store it but determineswhich version (stored record versus received flat file) is morecompletely filled out and will retain that copy. This process isdescribed in greater detail later herein in reference to FIG. 7.

[0088] In one embodiment, a cutoff date is defined after which nofurther submissions are accepted. In the case of mailed submissions, thepostmark date is the date used to determine timeliness of submission.Thus, further to this embodiment, an extension time period (such as 10days) is employed to allow timely mailed hardcopies and other mailedmedia to arrive at the competition system. Once it is determined thatall documents relating to candidate applications have been incorporatedinto the system, the qualification and eligibility subsystem (230)implements basic qualification and eligibility processes.

[0089] The qualification and eligibility subsystem (230) is showncomprising a completeness filter module (232), a qualification filtermodule (233), a partner filter module (234), and an eligibility filtermodule (235).

[0090] The completeness filter module (232) is coupled to thequalification filter module (233). The qualification filter module (233)is coupled to the partner group filter module (234). The partner groupfilter module (234) is coupled to the eligibility filter module (235).The completeness filter module (232), the qualification filter module(233), the partner filter module (234), and the eligibility filtermodule (235) are all coupled to both the non-select segment interfacemodule (222) and the candidate segment interface module (223).

[0091] The completeness filter module (232) (also called the level 1filter module) analyzes applicant packages for completeness. In oneembodiment, the completeness filter module (232) checks applicantpackages for both document completeness (i.e., that all requiredapplicant documents are present) and information completeness (i.e.,that all required data or information is present). Any applicant packagewhich fails either document completeness or information completeness istransferred to the non-select database and the applicant package isdeleted from the applicant list and entered in the non-select list. Theapplication of level 1 filters is discussed in greater detail laterherein in reference to FIG. 8.

[0092] The qualification filter module (233) (or level 2 filter module)analyzes candidate packages to determine if the candidates meet variousminimum qualification standards. By way of example, these minimumstandards can include a minimum GPA standard.

[0093] The partner group filter module (234) (also called the level 3filter module) analyzes the candidate packages to determine if thecandidates meet various partner group-specific minimum standards.Partner groups are subpools of candidates (i.e., less than the full poolof candidates in the competition system) based on any one or morecandidate-specific criteria. Partner groups are created so as to providethe ability to evaluate, rank, and select winners in ways sensitive tothe specific partner group. By way of example, partner groups might beset up in an academic scholarship setting by looking to such variablesas the candidates' geographical location, ethnic culture, poverty level,mother tongue, national origin, age, and so on. Thus, for a partnergroup for candidates having a non-English mother tongue, abilities inEnglish might not be used during filtering as having little or nocorrelation to the academic abilities of candidates in that partnergroup. Conversely, English abilities might be stressed in the filteringof partner groups for which English as a mother tongue was a criteria.

[0094] The eligibility filter module (235) (or level 4 filter module)imposes one or more heightened minimum response standards. In oneembodiment for a competition system having multiple optional essayquestions, for example, a minimum number of essay responses are requiredby the eligibility filter.

[0095] It is noted that the use of more than one filter level or the useof partner groups is not necessary and having all filtering done by onesoftware module is possible. The separation of filters into levelsallows for easier use of criteria having different purposes. Separationof filters also allows for better separation of criteria which is usefulwhen different individuals or committees are responsible for differentfilter criteria.

[0096] The operation of the level 2-4 filters is discussed in greaterdetail later herein in reference to FIG. 9.

[0097] The evaluation and selection subsystem (240) implementsevaluation of applicant packages and selection of competition winners.

[0098] The evaluation and selection subsystem (240) is shown having aqueue with reader module (241), a reader eligibility module (242), acandidate eligibility module (243), a scoring module (244), a readverification module (245), a phase 1 selection module (246), a phase 2selection or tie-breaking module (247), and a reader monitoring module(248).

[0099] The queue with reader module (241) is coupled with the readereligibility module (242). The reader eligibility module (242) is coupledto the candidate eligibility module (243) and the queue with readermodule (241). The candidate eligibility module (243) is coupled to thescoring module (244) and the non-select segment interface module (222).The scoring module (244) is coupled to the read verification module(245) and the reader monitoring module (248). The read verificationmodule (245) is coupled to the phase 1 selection module (246) and thequeue with reader module (241). The phase 1 selection module (246) iscoupled to the phase 2 selection or tie-breaking module (247). Thereader eligibility module (242), the candidate eligibility module (243),the scoring module (244), the phase 1 selection module (246), and thephase 2 selection or tie-breaking module (247) are all coupled to thecandidate segment interface module (223).

[0100] The queue with reader module (241) queues candidate packages withevaluators or readers for later evaluation (also called reading). In oneembodiment, each applicant package is matched and queued with oneevaluator. Matching is performed by comparing information from candidatepackages with potential evaluators. In one embodiment, candidatepackages are matched to evaluators by one or more of the followinginformation: cultural background, ethnic background, age, or geographicresidence. Each evaluator thereafter should have one or more candidatepackages listed in the queue for that evaluator.

[0101] The reader eligibility module (242) requires the evaluator topass an eligibility procedure for each candidate package prior toevaluation. This process requires the evaluator to answer one or moreevaluator/reader eligibility questions with respect to a particularcandidate package. The reader eligibility module (242) determines fromthe evaluator's responses whether the evaluator is able to evaluate thecandidate package or not. In one embodiment, for example, evaluatorquestions can be concerned with whether the evaluator attended the sameschool as the candidate or has other connections with the candidatewhich could raise questions of bias. If the evaluator is unable toevaluate the candidate, the evaluator is blocked from evaluating thecandidate and the candidate package is requeued with another evaluatorat the queue with reader module (241).

[0102] The candidate eligibility module (243) requires the evaluator todetermine whether the candidate is eligible. This determination can besimply a verification check of the same requirements enforced by thefilters in the qualification and eligibility subsystem (230) butadditional or alternative requirements can be applied by this module.

[0103] The scoring module (244) implements the actual evaluation ofapplicant packages by evaluators. An embodiment for an algorithmimplementing evaluation or reading of candidates is discussed in detaillater herein in reference to FIG. 11. The read verification module (245)verifies that the candidate package was evaluated by the required numberof evaluators. For example, in one embodiment, each candidate packagemust be evaluated by two evaluators, thus candidate packages havingfewer than two evaluations are requeued with another evaluator at thequeue with reader module (241).

[0104] The phase 1 selection module (246) handles initial selection ofcandidate finalists who, if confirmed at the confirmation subsystem(250) described hereinafter, become competition winners. The phase 2selection or tie-breaking module (247) handles the second phase or stageof candidate finalist selection. The phase 2 selection module (247)employs a finer selection paradigm than the phase 1 selection module(246) and is used to select between candidates who are very similar inoverall evaluation scores. An algorithm for the phase 1 selection module(246) is discussed in detail later herein in reference to FIG. 12a andan algorithm for the phase 2 selection module (247) is discussed indetail later herein in reference to FIG. 12b.

[0105] The reader monitoring module (248) implements monitoring of theevaluators during evaluations and can terminate an in-progressevaluation or nullify a completed evaluation if a problem is detected.Monitoring can take several forms. In one embodiment, evaluatorresponses are compared against corresponding reference standards. Forexample, if a competition involves evaluating candidates over severalcriteria such as writing skill and other criteria, monitoring willinvolve comparing the evaluator's responses for these criteria tostandards. The evaluator's response to writing skill, for example, canbe compared to the average score the evaluator has given to writingskill in all previous evaluations completed by that evaluator. Anothermethod is to compare the evaluator's responses to a predeterminedstandard or to the average across all evaluators. If an evaluator isfound to be erratic, corrective measures such as imposing a break,counseling, or so forth can be mandated. Additionally, completedevaluations found to be erratic can be nullified and the candidatepackage requeued with another evaluator.

[0106] The database interfacing subsystem (220) provides the necessaryinterfacing between the database hardware (104) and the rest of thesystem. The database interfacing subsystem (220) comprises a duplicatecheck/database interface module (221), a non-select segment interfacemodule (222), a candidate segment interface module (223), and acompetition-winner segment interface module (224).

[0107] The duplicate check/database interface module (221) is coupled tothe database hardware (104), the network interface module (211), themanual input module (212), the field extraction and OCR module (214),and the completeness filtering module (232). The duplicatecheck/database interface module (221) supplies and retrieves flat filesfrom the database hardware (104). Additionally, the duplicate checkportion of the duplicate check/database interface module (221)determines whether a flat file to be stored already exists in thedatabase hardware (104) as a record, and if so, determines which of thetwo are to be stored as the most complete version and which is to be“set aside” (i.e. stored but indicated as a rejected duplicate). Theduplicate check/database interface module (221) also, during the storingof flat files, determines if any other records in the database (104)relate to the same applicant as the file to be stored, and if so, theyare associated together. For example, in systems requiring an applicantpackage to consist of an applicant form, a nominator form, and arecommender form, the duplicate check/database interface module (221),when storing a document, will check if the other two forms are presentfor the form being stored, and if one or both is, the forms will beassociated together as an applicant package. An algorithm forimplementing the duplicate check/database interface module (221) isdiscussed in greater detail in reference to FIG. 7.

[0108] The non-select segment interface module (222) is coupled to thecompleteness filter module (232), the qualification filter module (233),the partner group filter module (234), the eligibility filter module(235), and the candidate eligibility module (243). The non-selectsegment interface module (222) interfaces other software modules withthe non-select segment (112) which holds candidate packages ofcandidates who ultimately do not become competition winners.

[0109] The candidate segment interface module (223) is coupled to thecompleteness filter module (232), the qualification filter module (233),the partner filter module (234), the eligibility filter module (235),the reader eligibility module (242), the nominee eligibility module(243), the scoring module (244), the read verification module (245), thephase 1 selection module (246), and the phase 2 selection ortie-breaking module (247). The candidate segment interface module (223)interfaces other software modules to the candidate segment (114) foraccess to candidate packages which have not been marked as non-selected.

[0110] The competition-winner segment interface module (224) is coupledto the phase 1 selection module (246), the phase 2 selection module(26), the confirmation module (252) (via the connector “C”), and there-qualification module (263) (via the connector “D”). Thecompetition-winner segment interface module (224) interfaces othersoftware modules to the competition-winner segment (116) for storage ofand access to candidate packages marked as competition winners.

[0111] Referring to FIG. 3b, shown is part 2/2 of a block diagram ingreater detail of the software architecture of FIG. 2.

[0112] Shown are a confirmation subsystem (250), a tracking andre-qualification subsystem and a payment and correspondence generationsubsystem (270). Each of these subsystems has various modules that arediscussed in greater detail as follows.

[0113] The confirmation subsystem (250) is shown having a confirmationdocument request module (251) and a confirmation module (252). Theconfirmation document request module (251) is coupled to theconfirmation module (252) and the correspondence printing interfacemodule (271). The confirmation module (252) is coupled to thecorrespondence printing interface module (271), the check printinginterface module (272) and the competition winner tracking module (261).The confirmation subsystem (250) implements confirmation of candidatefinalists.

[0114] The confirmation document request module (251) determines whatdocuments are needed to confirm a candidate finalist and handles thegeneration of correspondence to request these documents. Theconfirmation module (252) implements the actual confirmation ofcandidate finalists and handles the generation of award or rejectionletters and, in the case of winners, initial check issuance ifappropriate.

[0115] An exemplary algorithm for the operation of the confirmationsubsystem (250) is discussed in greater detail later herein with respectto FIG. 13.

[0116] The tracking and re-qualification module (260) is shown having acompetition winner tracking module (261), a re-qualification documentrequest module (262), and a re-qualification module (263).

[0117] The tracking and re-qualification module (260) implementsprogress tracking of competition winners and submits the competitionwinners to re-qualification to ensure that they continually meet anyrequirements of the competition system (100). For example, in oneembodiment for providing academic scholarships, competition winners(scholars) have defined academic requirements in order to continuereceiving scholarship award payments. The competition winner trackingmodule (261) is coupled to the re-qualification document request module(262). The re-qualification document request module (262) is coupled tothe re-qualification module (263) and the correspondence printinginterface module (271). The re-qualification module (263) is coupled tothe correspondence printing interface module (271), the check printinginterface module (272), the competition-winner segment interface module(224), and the document receiving module (215).

[0118] The competition winner tracking module (261) determines whenre-qualification or progress documents or information is needed. There-qualification document request module (262) handles the schedulingand printing of correspondence to the necessary recipients which requestthe submission of required documents or other information so thatcompetition winner tracking and re-qualification can be carried out. There-qualification module (263) implements the actual re-qualificationevaluation of competition winners.

[0119] An exemplary algorithm for the tracking of competition winners isdiscussed in greater detail later herein in reference to FIG. 14.

[0120] The payment and correspondence generation subsystem (270) isshown having a correspondence printing interface (271) and a checkprinting interface module (272).

[0121] The payment and correspondence generation subsystem (270)implements generation of checks and correspondence. The correspondenceprinting interface (271) is coupled to the confirmation document requestmodule (250), the confirmation module (252), the re-qualificationdocument request module (262), and the re-qualification module (263).The check printing interface module (272) is coupled to the confirmationmodule (252) and the re-qualification module (263).

[0122] The correspondence printing interface (271) interfaces withwordprocessors or other end devices such as the printer (150) to printcorrespondence necessary for implementation of any other processes inthe system. The check printing interface module (272) enables theprinting of competition award checks such as on the printer (152).

[0123] The reporting module (280) is shown with multipleinterconnections radiating outward indicating that it is coupled withsubstantially every other module shown in FIGS. 3a and 3 b. Thereporting module, as previously discussed, collects information andgenerates reports. As an example of the detail which the reportingmodule (280) collects, the reporting module (280) is coupled to thequeue with reader module (241) from which it determines each evaluatorthat an applicant package is queued to. From the reader eligibilitymodule (242), the reporting module (280) documents the results of thereader evaluation, and so on. From these data, the reporting module(280) can generate reports documenting exactly what occurred with aparticular applicant package as well as the performance of a particularreader, for example.

[0124] The external systems interface module (290) is likewise coupledto all modules necessary in FIGS. 3a and 3 b. As discussed previouslyherein, the external systems interface module (290) interfaces themodules and subsystems shown in FIGS. 3a and 3 b with external systemssuch as finance systems, human resources systems, and so forth.

[0125] Referring to FIG. 3c, shown is the block diagram of FIG. 3a andthe reporting module (280) showing interconnections of the reportingmodule (280). Similarly, referring to FIG. 3d, shown is the blockdiagram of FIG. 3b showing interconnections of the reporting module(280).

[0126] Shown in FIG. 3c are all of the modules shown in FIG. 3a and thereporting module (280). Shown in FIG. 3d are all of the modules of FIG.3b. The reporting module (280) is shown in both figures connecting withall the shown modules. In operation, the reporting module (280) recordsthe events of each module for documentation purposes. The reportingmodule (280) is able to synthesize a great variety of reports from therecorded information including, but not limited to document submissionerror reports, scanning error reports, possible duplicate reports, listsof all applicants for a specified competition, lists of all floaterdocuments, status reports on a specified applicant package or aspecified set of applicant packages (such as by an entered list or aspecified partner group or a group defined by a specified piece or rangeof information), evaluation summary reports by applicant, partner group,evaluator, scholar financial reports, scholar progress reports, dataanalysis reports, trend analysis reports, and so forth. The reportingmodule also stores and can reference in reports the documenting ofdecisions by competition system personnel. Such documentation providesproof, if later required, of the decisions made by system administratorsand personnel. Decisions which may be documented in this way includedecisions regarding duplicate document and fraud situations, the orderof evaluation variable application for tie-breaking, what constitutes acomplete document, and so forth. It is noted that one of ordinary skillin the art, in implementing a competition system (100) for a definedpurpose and a defined reporting capability, would know what modules tocouple to the reporting module and what data the reporting module wouldaccumulate and store. Additionally, one of ordinary skill in the art,given a defined purpose and a defined reporting capability, would knowwhich modules the external systems interface module (295) would need tobe coupled to and what transfer protocols would need to be implemented.

[0127] Referring to FIG. 4, shown is a general flowchart of the stepstraversed in operation of the systems of FIGS. 1 and 2.

[0128] The competition system (100) receives (402) application documentswhich can originate from applicants themselves and/or other individuals.In one embodiment, as discussed previously, are applications anddocuments in support thereof from nominators or recommenders. Thesesubmissions can be via electronic means such as, but not limited to,electronic submissions via the network (122) such as via the Internet,or via hardcopy submissions such as paper copies submitted by postalmail (130). In the case of hardcopy submissions using a preformattedform designed for the system of the present invention, the hardcopy isscanned using a scanning system (132). During the scanning process, formfields are identified and the information contained therein is extractedand read. In one embodiment, the scanning is done by optical means usinga commercially available scanner although any other method, such as bymechanical means, can be implemented.

[0129] In one embodiment, regardless of method of submission, forms anddocuments must be converted to a flat file format discussed previouslyin reference to FIG. 2. In submissions via the network (122), the flatfile is created directly from the inputted information. In submissionsincorporated via the scanning system (132), the scanning system producesthe flat file from the data retrieved during field extraction andoptical character recognition (OCR). Similarly, flat files are generatedfrom electronic submissions on computer-readable media or manual entryvia a workstation such as workstation (140).

[0130] Once the flat file form of received documents are present, thesystem stores (404) the flat files in the database hardware (104) asrecords and they are marked as being in the staging segment (110). Atthe time of storing, the system groups (404) or associates the flat filewith any other records from the same applicant. It is at this stage thatthe various records relating to the same applicant are associated toform applicant packages. While it is possible that duplicates of one ormore of these forms can be submitted to the system, only one of eachform type is included in an applicant package. Any duplicate forms areretained but identified as duplicates (the process of identifying theduplicate forms and excluding them from the candidate packages is termed“setting aside”).

[0131] In one embodiment, the association of records is done byapplicant social security number. In this embodiment, the applicantrecord (the application submitted by the applicant his- or herself) isthe key file (or key record). Once the applicant form is stored as arecord, the applicant identified in the record is added to an applicantlist. If any other required forms identifying that applicant are latersubmitted, such as a nominator form or a recommender form, they areassociated with the corresponding applicant record or key file. Requiredforms such as nominator forms or recommender forms, which are submittedwithout a corresponding key form (the applicant form) being present aretermed “floaters” indicating they do not have a key file to beassociated to.

[0132] After the cutoff date (406), further electronic submissions viathe network interface module (211) are prevented. The cutoff date ischosen so as to provide enough time to carry out the rest of thecompetition processing prior to any necessary deadlines. In oneembodiment, if hardcopy forms submitted by mail or other means areallowed in a competition, the postmark dates of mailed submissions areused to determine whether a submission was timely. When using postmarkeddates to determine cutoff timeliness, extra time is allowed beyond thecutoff date (such as 10 days) to allow for the physical delivery of themailed submissions to the competition processing site and the necessarysubsequent incorporation of the hardcopy data into the competitionsystem. All applicants identified in any form which was received by thecutoff date and the postal mail extension time period allowed for areidentified and recognized within the system and will therefore beconsidered during the competition.

[0133] At this point, the competition system checks (408) thecompleteness of the applicant packages. This determination is alsoreferred to as level 1 filtering. In one embodiment, the filtering isdone in two or more stages or levels which are described in greaterdetail in reference to FIG. 8 and FIG. 9. In one embodiment, thecompleteness of applicant packages is checked in two ways. Onedetermination is whether an applicant package has document completeness.As discussed previously in reference to one embodiment, a completeapplicant package is comprised of an applicant or nominee form, anominator form, and a recommender form. Any package determined not tocontain all required initial forms is determined to be incomplete. Asecond determination is whether each document in a candidate packagecontains all required data or information. As each document required ina complete candidate package is different from all others, each documentgenerally has different required data or information standards.Applicant packages which are lacking one or more required documents orwhich contain documents missing one or more required datum or pieces ofinformation are determined to be incomplete. These applicant packagesare redesignated as non-selects and are removed from the applicant list,added to the non-select list, and, along with any duplicate forms, aremarked to be in the non-select segment (112). All complete applicantpackages which meet the basic eligibility requirements are marked to bein the candidate database (114). Thereafter, these applicant packagesare referred to as candidate packages. Candidate package are thenfiltered (412) with one or more higher level filters.

[0134] After candidate packages pass the filtering process, they areread (414). Reading involves evaluation of the candidates with respectto the rubric of the competition system. The term rubric refers to thevariables and the scoring process which are used to score or select thewinning candidate. The variables can be objective or quantitative (suchas grade point average or GPA) or subjective or qualitative (such as theresults of essay evaluation). The variables can also include bothcognitive variables (GPA, essay writing, etc.) or non-cognitivevariables (leadership experience, presence of a support person, etc.).In one embodiment, evaluation or reading is done by humans who aretrained to evaluate the candidate packages, although automated readingimplemented by software means or a combination of automated processesand human evaluation can be implemented. In one embodiment, humanreaders evaluate the candidate packages against a rubric composed in thegreater part by non-cognitive variables such as, but not limited to:positive self-concept, realistic self-appraisal,understanding/navigation of social systems, preference for long-termgoals over short-term needs, availability of a support personage,leadership experience, community service, and non-scholastic knowledgeconcentration (i.e. appreciable knowledge acquisition about a field notcovered in past academic environments).

[0135] Evaluation or reading is generally done multiple times for eachcandidate package for reasons of reducing the effects of spurious readdeviations. In one embodiment, the reading process is completed twoindependent times for each candidate package (i.e., the candidatepackages are each evaluated by two different readers). In otherembodiments, however, multiple partial reads are used which allowsexpert readers to specialize on a subset of evaluation variables. Instill other embodiments, only one read or more than two reads can beused per candidate depending on the repeatable accuracy of readers, thelimitations of the competition finances, or the fairness safety factordesired. The process of evaluation or reading is described in greaterdetail later herein in reference to FIG. 10.

[0136] In order to ensure the accuracy and repeatability of human (andautomated) readers, reader normalization (or reader norming for short)is implemented in one embodiment. Reader normalization refers to thoseprocesses implemented to ensure readers operate within a predeterminedvariance standard during their evaluation. Reader normalization is usedduring training and continued during the actual reading of candidates.As a component of reader norming during actual evaluations, readermonitoring is implemented which allows for detection of errantevaluations or biases. Reader norming is discussed in greater detaillater herein in reference to FIG. 11.

[0137] Once the candidate packages have been read, the candidatefinalists are selected (416). Selection generally is implemented byselecting the candidates with the highest read scores. The selectionprocess can be complicated and involved when the number of competitionwinners is predetermined and the number of read variables is relativelylow because generally there will be a read score for which somecandidates will be selected, but due to the limited number of totalcompetition winners, others at the same score will not be able to beselected. The selection process is described in greater detail laterherein in reference to FIGS. 12a and 12 b.

[0138] Once the candidate finalists are determined (416), the candidatefinalists are put through a confirmation process (418) which involvesverifying statements made in the candidate's application (the nomineeform in an embodiment) and any other required documentation (such as thenominator and recommender forms, in an embodiment) and also verifyingthat the candidate meets any eligibility requirements the data for whichwere not known at the time of application. The process of confirmationis aided by the reception (420) of required confirmation documents. Byway of example, in an academic scholarship competition it may be thatthe candidate does not know which college or university he or she willattend at the time of application, but this information must be knownand supplied prior to the academic scholarship grant. The process ofconfirmation is discussed in greater detail later herein with respect toFIG. 13.

[0139] After confirmation, competition winners are tracked (422) tomonitor their progress and, optionally, to determine whether or not thecompetition winner is maintaining any post-competition requirements. Inan academic scholarship system, for example, post-competitionrequirements may relate to academic performance, and if met, the systemresponds by continuing payment of the scholarship award. Thepost-competition tracking system is discussed in greater detail laterherein in reference with FIG. 14.

[0140] Referring to FIG. 5, shown is a flowchart of an exemplaryalgorithm (500) for the steps traversed in the operation of electronicdocument submission of FIG. 4.

[0141] As one method of receiving information from potential candidates,a website is maintained which allows access and online informationsubmission by individuals. When an individual accesses (502) thewebsite, for security reasons, the website requires (504) identificationof the accessing individual by a logon protocol. Additionally, in oneembodiment, security is implemented during the submission process. Byway of example, digital signatures are used and retained by the systemfor verification. Alternatively, the system can require or allow the useof personal identification smart cards to verify identity.

[0142] Once the individual is successfully logged on, the system willprovide the individual with options which include online form completionand submission. The individual then indicates his or her role (506)(e.g., in one embodiment as applicant, nominator, or recommender) and,in the case of individuals providing support documents, the identity ofthe applicant (504) for which they wish to submit a form. Afterselecting online form creation, the system which first determines (508)whether that form was already submitted (i.e. whether that individualalready submitted a form for the candidate identified). Duplicatesubmissions are sometimes requested for various reasons such as theindividual wishes to change some information, the individual forgets theprior submission, and so on. In order to prevent creation of duplicatedocuments, the system prevents (510) the individual from continuing ifthe form requested has already been submitted to the system.

[0143] In one embodiment, duplicate forms are detected using thecandidate's social security number only. In another embodiment,potential duplicate applications are detected by the use of the lastname, the first letter of the first name, the date of birth, and thestate of birth and/or residence. In this embodiment, potentialduplicates identified by these criteria are included in a duplicatereport and system administrators will be able to access a specialinterface set up to allow simultaneous display of data from twoapplications so that the two potentially duplicated forms can be viewedtogether. Duplicates can arise which have alternate candidate socialsecurity numbers. This can occur in the case of a duplicate form beinginput with an erroneous candidate social security numbers orintentionally changed social security number (i.e., fraud).

[0144] Detection of both innocent and intentional duplication of formsis necessary for proper competition administration. Cases of apparentduplicates with different social security numbers do often occur and acommon situation is the case of twins. By way of example, it is notedthat in some Asian cultures, twins may receive identical ornearly-identical names and thus may appear to be a highly likely case ofduplication when in actuality it is not. In one embodiment, the systemadministrator reviews all possible duplicate situations prior to theinvolved documents continuing in the competition process. To facilitatethis, the system generates a possible duplicate report such as shown inFIGS. 22-23.

[0145] If the form requested was not previously completed, theindividual is presented (530) with the appropriate form for onlinecompletion. In the embodiment shown, the individual is interactingdirectly with the system, so the system must wait (530) for each fieldsubmission (usually by pressing of the enter key or the tab key). Inanother embodiment, forms can be completed by use of software such as adownloaded program or browser plugin and the results submitted atcompletion.

[0146] Next, the system determines (512) whether the submit button(indicating the individual has completed the form and wishes to submitthe form for storage). If the individual's action was not submission butinformation entry in a field, the system determines (514) whether theentered data meets basic validity criteria for that field and presents(516) any error message to the individual. By way of example, if thefield requires the birthdate of the candidate, the system may impose ageneral form requirement that the date be entered in the common mm/dd/yyform (i.e. two digit month numeral followed by two digit day numeralfollowed by a two digit year numeral). The system could also provide apresumed validity requirement by presuming invalid any birthday which isin the future, was in the past 10 years, or was prior to 100 yearspreviously (under the hypothesis that no candidate would be younger than10 years old or greater than 100 years old). If an error message ispresented, the cursor resets (518) in the field where the invalid datawas entered and the system again waits (532) for information entry andthereafter continues as discussed previously.

[0147] If a field submission is determined (514) valid, the systemautomatically positions (520) the cursor in the next field andthereafter waits (532) for further information entry and thereaftercontinues as described previously.

[0148] When the individual has completed the form and selects the submitbutton, the system determines (522) whether the form is completelyfilled out. If not, the system allows (524) the saving of incompleteforms for later continuance after which the online form entry session isfinished for now as shown by done indicator (526). If the form isdetermined (522) to be complete, the form is stored in the databasehardware (104) as a record and marked (528) to be in the staging segment(110) and the form entry session is finished as shown by done indicator(526).

[0149] Referring to FIG. 6 shown is a flowchart of an exemplaryalgorithm (600) of the steps traversed in the operation of hardcopysubmissions in the systems of FIGS. 1-2.

[0150] Another way besides electronic submission for forms to besubmitted by individuals is by paper or hardcopy submission such as bypostal mail0. Other forms of submission could also be used whichinclude, but are not limited to, facsimile submission, compact disc orother optical recording device submission, electronic file submissionother than online form completion such as by a portable document format(PDF) file, submission on magnetic media such as tape or disc (e.g.,floppy disc or harddisk), and mechanical submission such as by punchedcard, sheet, or tape.

[0151] As the competition system (100) receives (602) submissions, theyare collated (604) by form type (e.g. by applicant, nominator, andrecommender forms as discussed previously for one embodiment). Atperiodic intervals, or as convenient, the received forms are scanned(606) in batches. After scanning, the resultant images are submitted(610) to a field extraction and optical character recognition (OCR)process to extract the data in the images. The system then determines(612) whether the field extraction and OCR process was successful, andif it was, the algorithm (600) then creates (614) an export file andthen is done for the data extraction of that image as indicated by thedone indicator (616). After this, the algorithm (600) generally selects(602) another form for processing and continues as before until allscanned documents are likewise submitted to the field extraction and OCRprocess. If the field extraction and OCR process is determined (612) tohave not been successful, forms can be manually entered (618). Thealgorithm (600) then creates (614) an export file and then is done forthe data extraction of that image as indicated by the done indicator(616).

[0152] Generally, errors sometimes result in interpretation of fieldinformation. By way of example, an error could be as simple anoccurrence as a comma having been entered when a “/” was required (suchas in a date field). A scan report is generated which summarizes theresults of the scan process of each batch and if errors resulted, theseare indicated by document in the scan report.

[0153] Referring to FIG. 7, shown is a flowchart of an exemplaryalgorithm (700) of the steps traversed in the storing and grouping ofrecords shown in block (404) of FIG. 4.

[0154] The algorithm (700) stores flat files into the database hardware(104) as records. These flat files are received through any of themethods discussed previously in reference to FIGS. 5-6 (such aselectronically and via postal mail with subsequent scanning or manualentry). The algorithm (700) begins by determining (702) the form type(i.e. in one embodiment, whether the flat file represents a applicantform, nominator form, or recommender form) and applicant identity fromthe flat file. In one embodiment this is easily done as forms must besubmitted using predefined forms having specified fields or identifyinginformation or markings which identify the nature of the form. Next, thealgorithm (700) determines (703) whether any records in the database(104) for applicants with a different social security number have thesame form type and contain data which substantially matches the flatfile to be stored. If any records are found substantially matching thefile to be stored, this is a potential fraud situation and an entry ismade (704) in the duplicate list.

[0155] Next, the algorithm (700) determines (705) if a record exists inthe database (104) which is a match for the form type and the sameapplicant. If not, the flat file is stored (706) in the databasehardware (104) as a record. If the form is an applicant form (708), itis indexed (710) in the applicant list or key list. The applicant formor key list is The algorithm (700) then checks a floater list for anyfloaters corresponding to the stored applicant record and if there areany, they are associated (712) with the just-stored applicant record.The floater list is a list of all forms (such as the nominator andrecommender forms in the prior embodiment) for which a correspondingapplicant record has not been stored in the database hardware (104). Thealgorithm (700) is then done for this flat file as indicated by the doneindicator (714).

[0156] If the flat file is not an applicant form (708), a check (716) ismade if the corresponding applicant (or key) form has already beenindexed. This is done in one embodiment by use of the candidate's socialsecurity number (SSN), which is also required on the nominator andrecommender forms in identifying the applicant for which the form issubmitted. If a corresponding applicant form has not been received bythe competition system (100), the file is indexed (720) and flagged as afloater in the floater list. After indexing of the file in the floaterlist, the algorithm (700) is done with this form as shown by the doneindicator (714).

[0157] If a corresponding applicant form is found (716) to have beenindexed by the system, the stored record is associated (718) with thecorresponding applicant or key form. The algorithm (700) is then donewith this form as shown by the done indicator (714).

[0158] If the flat file is determined (705) to already be in thedatabase hardware (104), this means that the flat file is considered apossible duplicate and is indexed in a duplicate list. The duplicatelist is a report which keeps track of all duplicates. The algorithm(700) next compares the stored record with the received flat file todetermine (722) whether the received flat file is more complete than therecord already stored in the database hardware (104). If it is, therecord already stored in the database hardware (104) is flagged as “setaside” and the received file is stored and is the record which is usedby the competition system (100) during further considerations.

[0159] It is noted that as discussed previously, by consideration of theduplicate list, system administrators or other officials can verifywhether the correct action was taken.

[0160] If the received file is determined (722) to be less complete thanthe record stored already, the received form is stored (724) in thedatabase hardware (104) and flagged as “set aside”, but is stillassociated (726) with the other documents of the same applicant. Thealgorithm (700) is then done with this form as shown by the doneindicator (714).

[0161] Referring to FIG. 7a, shown is a flowchart of an exemplaryalgorithm for duplicate detection review in one embodiment.

[0162] In one embodiment, system administrators or other authorizedpersonnel, must review all possible duplicate detections prior to thequalification and eligibility phases. A reviewer accesses the possibleduplicate list to determine the possible duplicate situations to review.After selection (752) of a possible duplicate situation to review, thereviewer notes (754) the nature of the situation. If the situation isone of two or more documents identifying the same applicant socialsecurity number (SSN), the reviewer reviews the involved documents todetermine which document is more completely filled out. In oneembodiment, the duplicate detection/database interface module (291)makes an initial decision as to which document is more complete. In thisembodiment, if the reviewer determines (756) that the duplicatedetection/database interface module (291) made the right decision,documents (758) that decision and is done for that situation. Otherwise,if the reviewer determines (756) that the decision of duplicatedetection/database interface module (291) was in error, the reviewercorrects the decision and documents (758) the decision and action and isdone for that situation.

[0163] If the situation is determined (754) to be a situation which doesnot involve identical social security numbers, the reviewer notes (762)whether it is a situation of possible fraud. Possible fraud situationsresult from two or more documents bearing demographic and otherinformation which appears to refer to an identical individual but whichidentifies different social security numbers. If the situation is one ofpossible fraud, the reviewer reviews (764) the involved documents andmakes a determination. Possible determinations include (1) that theinvolved documents relate to different individuals (i.e. no fraud), (2)that the documents refer to the same individual (i.e. actual fraud), (3)that the documents refer to the same individual but that one or moresocial security numbers were improperly entered, and so forth. After thedetermination and resultant action, the reviewer documents (766) thedetermination and action and is thereafter done (760) for the situation.

[0164] If the reviewer determines (762) that the situation is somethingother than simple duplicate detection or possible fraud, the reviewerreviews (768) the situation, makes a determination, implements thedetermination, and documents (720) the determination and implementation.Thereafter, the reviewer is done (760) with the situation.

[0165] Referring to FIG. 8, shown is a flowchart of an exemplaryalgorithm (800) of the steps traversed in the determination of candidatepackage completeness in FIG. 4.

[0166] Once the cutoff date (406) and any extended time for postaldelivery has been reached, applicant packages are analyzed forcompleteness. In one embodiment, applicant packages are analyzed forboth document completeness and each document is analyzed for datacompleteness. The algorithm (800) begins by selecting (802) an applicantpackage for analysis. The algorithm (800) determines (804) first whetherthe application package has data completeness, and if so, the algorithm(800) selects a form from the package. Applicant packages, to havedocument completeness, must be made up of a minimum required set ofdocument records. For example, in one embodiment, a candidate packagerequires an applicant form, a nominator form, and a recommender form tobe complete. The algorithm (800) then determines (808) whether the formhas data completeness. For an applicant package to have datacompleteness, each document or record in the applicant package must meeta minimum data completeness standard (such as all required fields havingnon-null entries or all required data present). Note that because eachdocument in an applicant package is different, each may have differentminimum data completeness requirements. If the record selected does havedata completeness, the algorithm (800) determines (810) whether any morerecords in the applicant package need to be checked for datacompleteness, and if so, the algorithm (800) selects (806) anotherrecord and continues as described previously. If there are no furtherrecords to be analyzed (810), the algorithm (800) is done for thispackage as indicated by the done indicator (812).

[0167] If a record is determined (808) not to have data completeness, itis flagged (814) as incomplete and the applicant package is marked (816)to be in the non-select segment (110). Thereafter, the algorithm (800)is done for this package as indicated by the done indicator (812).

[0168] If the applicant package is determined (804) to be lacking in oneor more records, the applicant package is flagged (818) as incompleteand designated (816) as a non-select by being marked as in thenon-select segment (110) and being listed in the non-select list.Thereafter, the algorithm (800) is done for this package as indicated bythe done indicator (812).

[0169] Referring to FIG. 9, shown is a flowchart of an exemplaryalgorithm (900) of the steps traversed in the level 2-3 filtering ofcandidate packages shown in FIG. 4.

[0170] After the candidate packages have passed the level 1 filters ofdata completeness and form completeness, they are filtered by one ormore higher level filters. In one embodiment, a candidate package isfiltered (902) by level 2 filters which ensure candidates meet certainqualifying characteristics. Further to this embodiment these qualifyingcharacteristics include a minimum GPA score or completion of a generalequivalency degree (GED) and minimum financial requirements (such as amaximum family income to be eligible for receiving an academicscholarship). The algorithm (900) then determines (904) whether thelevel 2 filter qualifications are met and if they are not, the candidatepackage is marked (906) to be in the non-select segment (112) and thecandidate is listed in the non-select list.

[0171] Otherwise, if the algorithm (900) determines (904) that the level2 filter qualifications are met, further filters can be applied to thecandidate package. In one embodiment, a candidate package is filtered(908) by level 3 filters which ensure that the candidate meets certainheightened response requirements. Further to this embodiment, the level3 filters are applied specific to the candidate's subgroup or partnergroup. In this embodiment, once candidates pass the level 1 and level 2filters, they are grouped together into two or more subgroups or partnergroups. In one embodiment, the grouping is done by the creation ofpartner lists. The grouping can be done based on any one or more piecesof data. By way of example, grouping could be done in an academicscholarship setting by looking to such variables as the candidates'geographical location, ethnic culture, poverty level, mother tongue, andso on. The level 3 filters are then be applied by subgroup/partner groupand ensure minimum qualifications designed in light of the specificcharacteristics of the particular subgroup.

[0172] In one embodiment, group or partner administrators are able toapply the level 3 filters more than one time in order to reduce thecandidate pool for that candidate group or partner group topredetermined levels selected as optimum prior to going into the read orevaluation process. Criteria applied during level 3 filtering includes,in one embodiment, a minimum number of academic awards, a minimum numberof public awards, a minimum number of honors, a minimum number ofleadership roles, a minimum number of student excel bubbles completed, aminimum class rigor average, a minimum amount of community service, amaximum amount of personal circumstance exceptions, and a minimum numberof paid hours of employment.

[0173] In another embodiment, the level 3 filters, in order toeffectively reduce the pool of candidates to a desirable number, arepredetermined. In this embodiment, prior to the start of the competitionprocess, a predetermined set of discretionary criteria and an orderingof their application is determined. Additionally, a desired pool size ora target range within which the pool must number is selected.Thereafter, in the actual filtering process, the discretionary criteriais applied in increments and the number of applicants remaining in thepool is ascertained after each increment, with the filtering stoppingwhen the target pool size is reached. By way of example, the first 10discretionary criteria in one embodiment are, in a competition foremployment at a law firm, a law degree from a 2^(nd) tier school orhigher, a grade point average of 3.2 or higher in a law degree,completion of one or more engineering degrees, a grade point average of3.2 or higher for one engineering degree, the attendance of a top 20engineering school for one engineering degree, a law degree from a1^(st) tier school, a grade point average of 3.5 or higher in a lawdegree, completion of two or more engineering degrees, a grade pointaverage of 3.5 or higher in an engineering degree, and the attendance ofa top 10 engineering school for one engineering degree. Thus, thesecriteria would be applied in order until the pool of applicants wasreduced to the desired level.

[0174] In another embodiment, the level 3 filters, in order toeffectively reduce the pool of candidates to a desirable number, aredetermined as a function of the number of applicants in the applicantpool. In this embodiment, the discretionary criteria is predeterminedbut instead of their application being dependent on a required outcome,their application is dependent on the initial size of the applicantpool. In this variation, the criteria itself can be dependent on thesize of the pool of candidates. For example, again in a system for theselection of a law firm associate, one criteria for a pool of applicantsbelow 50 is that the applicants must have a 3.2 grade point average orhigher in a law degree and for a pool of 100 or more candidates, thecriteria is a minimum of a 3.5 grade point average. Alternatively, thecriteria could be that the applicants must have a law school grade pointaverage which exceeds 4−(1/x) where x is the number of applicants in thepool. After either of the block (910) or the block (906), the algorithm(900) is finished as shown by done indicator (912).

[0175] If the candidate package passes (910) the level 3 filters, thecandidate package has passed the filtering phase of the competition asshown by the done indicator (912). If the candidate package does notpass (910) the level 3 filtering, it is stored (906) in the non-selectdatabase (112) and the candidate is listed in the non-select list.

[0176] Additional or fewer levels of filtering can be applied within thescope of the present embodiment. In one variation, an additional level 4set of filters is applied which ensure that the candidate has respondedto a minimum number of essay questions in his or her applicant form. Thesystem applies the level 4 filters in a manner similar to that asdescribed for the level 2 and level 3 filters.

[0177] Referring to FIG. 10, shown is a flowchart of an exemplaryalgorithm (1000) of the steps traversed in operation of the readingprocess in FIG. 4.

[0178] The system begins by queuing (1002) a applicant package with anevaluator (also referred to as a reader) who has not previouslyevaluated that applicant package or been disqualified from evaluating orreading that applicant package. The evaluator then begins by responding(1004) to a set of reader qualification questions which document whetherthe evaluator has any connections or conflicts with the applicant whichwould raise questions of bias. By way of example, in one embodiment,reader qualification questions include whether the evaluator served asnominator or recommender for the applicant and whether the evaluator hasany connection with the applicant such as having attended the same highschool or college as the applicant or that the evaluator has anypersonal reason not to read the applicant. In this example, a yes to anyreader eligibility question disqualifies the reader from reading theapplicant.

[0179] The system then checks (1006) whether the reader is qualified toread the applicant, and if not, the reader is disqualified (1008) fromreading the applicant and the system again queues (1002) the applicantpackage with another available reader.

[0180] After verification (1006) of reader eligibility to evaluate orread the applicant package, the evaluator verifies (1010) whether theapplicant is eligible based on the requirements of the competitionsystem. To facilitate this, the system presents the necessaryinformation extracted from the applicant package on the reader's displayalong with a statement of applicant eligibility requirements. The systemthen determines (1012) from the reader's verification whether theapplicant is eligible and if the applicant is not eligible, the systemmarks the applicant package to be in the non-select segment (112) andindexes the applicant in the non-select list.

[0181] If the applicant is determined (1012) to be ineligible forreading, the algorithm (1000) marks (1014) the applicant package to bein the non-select segment (112) and lists (1014) the applicant in thenon-select list. In this case, the algorithm (1000) is thereafterfinished as shown by the done indicator (1020).

[0182] If the applicant is (1012) eligible, the reader then reads (1016)the applicant package. Reading is facilitated by the side-by-sidedisplay of the read variable scoring area and an information display.FIGS. 20-21 show views of an exemplary display for reading. In theinformation display, the system can display applicant data segmentswhich are sections of data extracted from one or more forms in theapplicant package or the system can display complete forms. The systemalso blocks information which is of a personal nature such as theapplicant's social security number (SSN), email address, mailingaddress, alien registration number, date of alien registration numberissuance, and family information such as gross annual income, familysize, and number of family members currently in college. When the readerclicks or selects a variable, the system will pull up the relevant datasegments or applicant package forms and displays them in the informationarea for use by the reader.

[0183] The read variables are determined by the administrators of thecompetition system and the scores achieved by applicants on the readvariables are what determines whether they are ultimately successful inthe competition. By way of example, the read variables for oneembodiment in the academic scholarship arena include the non-cognitivevariables 1) positive self-concept, 2) realistic self-appraisal, 3)understanding/navigation of a social system, 4) preference of long-termgoals over short-term goals, 5) availability of a strong support person,6) leadership experience, community service, 7) interest/knowledge in anon-school field, 8) community service, as well as the cognitive-relatedvariables 9) curriculum rigor, 10) academic achievement, 11) essayresponse.

[0184] In one embodiment, the scoring during evaluation is by anumerical scale as follows: 1 indicates a negative score, 2 indicates aneutral score, 3 indicates a positive score, and 4 indicates anexcellent score. In some variable cases, less than the total number ofscores will be available. By way of example, in the case of thenon-cognitive variable presence of a strong support person, only the twoscores of 2 or 3 are allowed.

[0185] After the reader has completed and submitted the reading of theapplicant package, the system determines (1018) whether the applicantpackage has been read by the required number of different readers. Useof multiple readings has many benefits including, but not limited to,ensuring greater fairness and providing compensation for the event ofone reader providing a randomly “unfair”, inequitable, or inaccurateread which is not detected by the system. In one embodiment, twoindependent reads are required but more reads can be used within thescope of the invention. If the required number of readers has not readthe applicant package, the system queues (1002) the applicant packagewith another available reader and continues as before. Otherwise, thereading phase for the applicant package is complete as shown by the doneindicator (1020).

[0186] Referring to FIG. 11, shown is a flowchart of the steps traversedin the operation of reader selection and training for participation inthe algorithm (1000) of FIG. 10.

[0187] First, the number of readers is determined (1102). In oneembodiment, this is done manually by the system administrator based onthe number of candidates anticipated and either (a) the number of readsper reader desired or (b) the length of the reading phase considered inconjunction with the average number of reads per reader per day. Oncethe number of desired readers is determined, reader applications arecollected (1104) and evaluated (1106). In one embodiment, readers arefiltered to meet minimum criteria related to the nature of thecompetition to be conducted. By way of example, for a competition systemfor military officer selection, readers would likely be required to haveminimum qualifications which include military experience or familiarity,officer selection experience, and so forth. The administrators of thesystem then select (1106) the readers.

[0188] The readers are then trained (1108) with the goals and procedureof the competition system. Following the initial orientation, thereaders are trained (1110) about the read variables and the scoringrubric. In training the readers on the scoring process, the readers arefirst trained (1112) by paper and pencil scoring exercises whichfacilitates a personal review procedure whereby trainers may personallydiscuss the scoring rubric with the reader trainee. Once the readershave demonstrated a basic competence during the paper exercises, thereaders are then trained (1114) by computer-based exercises whichsubstantially mimic the actual reading displays and use actual priorcandidate packages (without personal identifying information). As withthe paper exercises, reader trainers are available to reinforce thescoring rubric by discussions of the methodology and balancing offactors which go into scoring the read variables. In closing theelectronic reading exercises, an evaluation is conducted (1116) toensure that the readers are reading at the level mandated by thecompetition system requirements. After the readers pass the evaluation,the reader training phase is finished as shown by the done indicator(1118).

[0189] Referring to FIGS. 12a-12 b, shown is a flowchart of an exemplaryalgorithm (1200) of the steps traversed in operation of finalistselection in FIG. 4.

[0190] Referring to FIG. 12a, shown is part 1/2 of the algorithm (1200).Finalist selection occurs (1202) after the reading is finished. Thenumber of competition winners to be chosen is read (1204) (oralternatively input by a system operator) and the read variable scoresfor each candidate are summed (1206) to produce read variable sums. Byway of example, in one embodiment, 11 read variables are scored eachscored by two evaluators. Thus, for each candidate, there are 22 totalread variable scores which are then summed together to provide the readvariable sum for that candidate. After summing the read scores, softwarevariables to be used in later processing are initialized (1208). Thecandidates are then grouped (1210) by read variable sums. By way ofexample, if 25 candidates out of a pool of 400 candidates achieve a readvariable sum of 80, then these 25 candidates form one group and theother 375 candidates form one or more other groups as dictated by theirread variable sums.

[0191] Once the candidates are clustered into groups by read variablesums, the groups are ranked (1212) from highest read score sum to nextlowest read score sum and so on until all groups are ranked. At thispoint, the actual selection process begins by first selecting (1214) thehighest ranking group as the active group. Next, the algorithm (1200)determines (1216) whether the total number of candidates alreadyselected (zero at this point) plus the number of candidates in theactive group exceeds the total number of competition winners to beselected. If the total number of candidates already selected (again,zero at this point) plus the number of candidates in the active group isless than the total number of competition winners to be selected, allcandidates in the active group are selected (1218). Selection entailsthe candidates being designated as finalists by a entry being made inthe finalist list and their candidate packages being flagged as afinalist. The algorithm (1200) then returns to block (1214) and selects(1214) the next highest ranking group as the active group and continuesas described before except that now, the variable describing the totalnumber of candidates already selected is no longer zero. The loopconsisting of the steps described, (1214) to (1218) inclusive, continuesuntil the first occurrence where the total number of candidates alreadyselected plus the number of candidates in the active group is determinedto exceed (1216) the total number of competition winners to be selected.At this point, the algorithm (1200) leaves the loop, steps (1214) to(1218), and continues (via the connector “C” (1220)) to the second levelor phase 2 of selection which begins at the step (1222) shown in FIG.12b.

[0192] Referring to FIG. 12b, shown is part 2/2 of the algorithm (1200).This part of the algorithm (1200) is an exemplary embodiment fortie-breaking. Tie-breaking refers to the method of selecting a subset ofan active group of candidates in order to end up with the same number ofselected candidates (i.e. finalists) as the total number of competitionwinners to be selected.

[0193] The embodiment shown in FIG. 12b uses 11 read variables duringthe tie-breaking process. However, part 2/2 of the algorithm (1200) asshown in FIG. 12b can be used with any number of read variables. Theblock (1222) is a loop control which limits the number of iterations toa maximum equal to the number of read variables available (11 readvariables in the shown embodiment). First, the algorithm (1200)determines (1224) whether the number of selected candidates (finalists)exactly equals the total number of competition winners to be selected,and if it does, the selection process is finished as indicated by doneindicator (1226). Otherwise, the algorithm (1200) selects (1228) thehighest priority read variable which was not already selected and reads(1230) the weight to be applied to that read variable.

[0194] Read variable priority is a ranking or scaling of the variousread variables for use during tie-breaking. The read variable prioritiesare, in an embodiment and for reasons of fairness, determined prior tothe reading process. In one embodiment, read variable priorities aredetermined by partner group and not globally over the entire pool ofcandidates. In this embodiment, the partner groups are free to determinethe read variable priorities in ways which makes the most sense relativeto the characteristics of the candidates of that partner group. The readvariable having the highest priority will be used first in anytie-breaking which needs to be done. Then, if the total number ofselected candidates is not yet reached, the next highest variable isused and so on until the total number of allotted awards are matched toselected candidates.

[0195] Once the highest priority read variable is determined, it isweighted (1232) by a predetermined amount. Further to the lastembodiment, the predetermined amount, like the priority ordering of theread variables, is determined before the reading process. Weightingrefers to multiplying the selected read variable by the predeterminedweight for each candidate in the active group. By way of example, for acompetition system applying two reads over 11 variables for eachcandidate, there are two read variables for each candidate which areweighted and the predetermined weight to be applied during first roundtie-breaking is 2. After weighting the read variables, a new readvariable sum is computed (1234). Next, the candidates (of only theactive group) are first clustered (1236) into subgroups or cells suchthat (a) each subgroup has candidates all having the same new readvariable sum (which incorporates the weighted variables) and (b) allcandidates having the same new read variable sum are in the samesubgroup. Next, the subgroups are ranked (1238) by from highest weightedread variable sum progressively down to the lowest weighted readvariable sum.

[0196] On the first pass through the tie-breaking loop (block (1222) toblock (1248) inclusive), only one read variable weight is applied. Oneach successive iteration through the tie-breaking loop another readvariable is weighted. Read score sums in a given iteration through thetie-breaking loop incorporate the weighted read scores of all previousiterations (e.g. once a read variable is weighted, the weighted variableis used for the rest of the tie-breaking rather than reverting to theoriginal non-weighted score).

[0197] The tie-breaking selection process now continues similarly to theselection process as described in reference to FIG. 12a. The highestsubgroup is designated (1240) as the active subgroup. Next, thealgorithm (1200) determines (1242) whether the total number of selectedcandidates plus the candidates in the active subgroup exceeds the totalnumber of competition winners to be selected. If not, the candidates inthe active subgroup are selected (1244) and the algorithm (1200)continues back and selects (1240) the next highest ranked subgroup asthe active subgroup and continues as before. The act of selecting of acandidate means that the candidate has been converted to finaliststatus. At this point, all that is done is that the candidate is indexedinto a selected list. The loop consisting of block (1240) to block(1244) is iterated until the total number of selected candidates plusthe candidates in the active subgroup is determined to exceed (1242) thetotal number of competition winners to be selected. When this occurs,the highest ranking subgroup having candidates who have not beenselected is designated (1246) the active group. Next, the algorithm(1200) checks (1248) whether the total number of competition winners tobe selected equals the number of selected candidates and if it has,algorithm (1200) has finished the selection process (as shown by block950). Otherwise, the algorithm (1200) continues back and determines(1222) the next highest priority read variable and continues as before.

[0198] Generally, this embodiment of tie-breaking is very effective innarrowing out of a group of candidates the necessary subgroup to fillout the total number of candidates to be selected. The only complicationis when two or more candidates receive exactly the same score on eachread variable and also just happen to fall within the read scorehierarchy such that only a subset of them can be chosen to fill out theallotment of awards. In this case, successive weighting and ranking willnot be able to differentiate between them so as to fill out the awardallotments. This situation can be handled by having backup variableswhich normally are not used for scoring and ranking which can be used.These backup variables could also be scored by the readers, oralternatively, they could be taken from candidate information such asGPA, normalized GPA, etc.

[0199] Once all of the finalists have been selected, the systemimplements the actual conversion of the candidates to finalists bydesignating all candidates as a finalists in the their respectivecandidate packages, adding the candidates to the finalist list, anddeleting the candidates from the nominee list.

[0200] It is noted that in one embodiment, candidate finalists aresubject to confirmation and if confirmed, are required to accept priorto being confirmed as a competition winner. In this embodiment, somecandidate finalists will not be confirmed and others who are confirmedmay not accept. Thus, in order for the predetermined number ofcompetition winners to be met, the system must select new competitionfinalists for each competition finalist who is ultimately not confirmedor who does not accept even when confirmed. In one embodiment, thesystem selects additional finalists by continuing from where the phase 1and phase 2 selection systems left off. The selection of additionalfinalists can be done periodically, such as once a week, or can be donemore frequently such as once a day or even each time notification ofnon-acceptance or non-confirmance is received.

[0201] Referring to FIGS. 12c- 12 e, shown is an example of anapplication of tie-breaking. Referring to FIG. 12c, shown is an exampleof the priority ordering of evaluation variables for two partner groupsin one embodiment.

[0202] As an example of tie-breaking, one embodiment concerns theselection of individuals for employment. In this embodiment, thevariables over which the candidate packages are evaluated are curriculumrigor, overall academic achievement, positive self-esteem, realisticself appraisal, understanding and navigation of social andorganizational systems, preference for long-term goals over immediategratification, leadership experience, community service, self-motivatedacquisition of knowledge or skills, persuasiveness, language structureand expression ability. These variables are chosen to maximize thepredictive ability of the competition system and it is noted that incompetitions employed for other purposes or with different goals, thesevariables may be chosen to differently.

[0203] In this embodiment, applicants are divided into partner groupswhich include partner groups for engineering and business management.Within the partner group for engineering the evaluation variables areordered for priority as: overall academic achievement, curriculum rigor,language structure and expression ability, preference for long-termgoals over immediate gratification, self-motivated acquisition ofknowledge or skills, understanding and navigation of social andorganizational systems, leadership experience, persuasiveness, positiveself-esteem, realistic self appraisal, and community service. Incomparison, within the partner group for business management theevaluation variables are ordered for priority as: understanding andnavigation of social and organizational systems, leadership experience,positive self-esteem, language structure and expression ability,persuasiveness, preference for long-term goals over immediategratification, self-motivated acquisition of knowledge or skills,overall academic achievement, curriculum rigor, realisticself-appraisal, and community service. It is noted these orderings perpartner group reflect the goals of the particular competition and wouldlikely be chosen differently for different competition objectives.

[0204] Referring to FIG. 12d, shown are an example of the evaluationresults for two candidates in an embodiment.

[0205] As an example, two candidates are in competition for anengineering position. As part of the qualification and eligibilityprocess, each was determined to have the necessary engineering degreesand so on. During the reading process, candidate # 1 received for reader#1: 4, 4, 2, 3, 3, 2, 4, 2, 3 and, for reader #2: 3, 4, 2, 2, 4, 2, 3,4, 4, 3, 2. Candidate #2 received for reader #1: 4, 4, 3, 3, 3, 2, 4, 4,3, 2, 2 and for reader #2: 2, 4, 4, 4, 3, 2, 4, 2, 2, 3, 2.

[0206] Referring to FIG. 12e, shown is an example of tie-breakingbetween the two candidates of FIG. 12d according to the evaluationvariable priorities of FIG. 12c.

[0207] Thus, in phase 1 of the selection process, both candidates havean overall score of 66 and so a decision of which to choose cannot bemade. In a situation where only one of these candidates is to beselected, resort is made to the phase 2 or tie-breaking selectionprocess which uses evaluation variable priorities. For the engineeringpartner group, the first priority variable is overall academicachievement (the second variable in the global listing of variables). Inthe example, both candidates have the values of 4, 4 in their tworeadings. If the weighting is 2, for example, these scores become 8 and8 for both candidates and their overall scores become 74 for both. Thesecond priority variable for the engineering groups is curriculum rigor(the first variable in the global evaluation variable listing).Candidate #1 received scores of 3, 3 for this variable whereas candidate#2 received 4, 2. Applying a weighting of 2 again, these scores become6, 6 and 8, 4, respectively and both candidates receive an overall scoreof 82. The third priority variable for the engineering partner group islanguage structure and expression ability (the last variable in theglobal evaluation variable order) for which candidate #1 received 3, 2and candidate #2 received 2, 2. Thus, applying a weighting of 2 for thisvariable results in scores of 6, 4 and 4, 4, respectively. Thus,candidate #1 now has a total score of 87 and candidate #2 now has ascore of 86 and the tie has been broken in favor of candidate #1.

[0208] Referring to FIG. 13 shows a flowchart of an exemplary algorithm(1300) of the steps traversed in operation of competition finalistconfirmation shown in FIG. 4.

[0209] competition finalist confirmation refers in part to theverification of various data presented on candidate package forms and inpart to verify the eligibility of the candidate relating to any newinformation such as may now be available concerning the tuition needs ofthe candidate finalist. In one embodiment, once candidates have passedthe reading and selection phases, they are transferred to a scholartracking module. Entry into the scholar tracking module allows forindividual access by administrators as various confirmation documentsarrive and are incorporated into the system and also allows manualupdate of information and candidate status. The scholar tracking systemallows for the administrator to search by social security number (SSN),partial SSN, specific surname, and generic surname. The term genericsurname refers to the entry of an initial set of letters followed by awildcard. The result of a generic surname search is a list of all namesmatching the query. By way of example, the query “Mus*” would result ina list containing Musam, Muse, and Musni provided those names were inthe database queried.

[0210] The algorithm (1300) causes (1302) correspondence to be sent tothe candidate finalists which announces their finalist status and whichrequests all necessary confirmation documents. In one embodiment, therequired confirmation documents include a complete high schooltranscript (covering at least the last 3 years attended) or a generalequivalency degree (GED) as appropriate, any tribal documents if thefinalist is Native American (note that some Native American tribes haveonly one document to indicate tribal membership whereas others may havemultiple documents which together indicate tribal membership), a UnitedNegro College Fund (UNCF) Scholars information form (discussed in detailin the next paragraph), a copy of the finalist's college/universityadmissions letter indicating admittance, a letter/document for eachscholarship (outside of this competition) which the finalist has beenawarded indicating the amount of award, and any documents for each loanthe finalist has indicating the amount and terms of the loan. In avariation of this embodiment, confirmation documents additionallyinclude documents attesting to citizenship or primary residency,affidavits and documents regarding community service, documentsregarding employment, documents regarding honors and awards, andaffidavits and documents regarding leadership experience.

[0211] The UNCF Scholars information form is a form developed tostandardize institution information disclosure relating to financialgrants to students. It has been determined through actual experiencethat the levels of detail provided by colleges and universities inresponse to inquiries concerning financial grants to students can varywildly from institution to institution. For example, some universitiesand colleges do not report 3^(rd) party scholarships in studentfinancial reports. In response to this, a standardized form wasdeveloped which specifies what information needs to be disclosed. In oneembodiment, the UNCF Scholars information form requires disclosure ofall scholarships, grants, loans, and other financial assistance to thestudent, both from the academic institution as well as 3^(rd) parties.This level of detail is needed, in this embodiment, as the goal of thecompetition system in which it is used is to (1) complete the financialneeds of competition winners by providing any required academic tuitionnot met by financial assistance already in place and additionally torepay in full all loans (so the student will not have that obligationlater).

[0212] Once confirmation documents are received, they are incorporatedinto the system by the document receiving subsystem embodiments of whichare similar to the procedure described in reference to FIGS. 4 and 5except that documents are stored (1304) in the candidate database andlinked immediately with the rest of the corresponding candidate package.

[0213] In the actual confirmation process, a candidate package isselected (1306). The algorithm (1300) then determines (1308) whether allrequired documents are present. If all the required confirmationdocuments are present, the candidate confirmation documents areevaluated (1310). In one embodiment, the evaluation of candidateconfirmation data is done by a system administrator. The algorithm(1300) then determines (1312) whether the candidate has been confirmed,and if so, the candidate is converted (1314) to scholar status.Conversion to competition winner status includes the marking of thecandidate package as being in the competition-winner segment (116) andthe addition of the candidate to a competition winner list. Thecompetition winner list is the master list of all competition winnersfor use by the system. From this point on, the candidate is referred toas a competition winner and his or her package is referred to as acompetition winner package.

[0214] After conversion (1314) of a candidate to a competition winner,the algorithm (1300) determines (1316) whether any more candidateconfirmations are outstanding, and if so, the algorithm (1300) selects(1308) another candidate package and continues as before. If thealgorithm (1300) determines (1316) that no candidate packages areoutstanding, the confirmation process is finished as shown by the doneindicator (1316).

[0215] If the algorithm (1300) determines (1308) that one or morerequired confirmation documents are missing, the candidate is converted(1320) to a non-select by marking the candidate package as beingnon-select and listing the candidate in the non-select list. Thereafter,the algorithm (1300) determines (1316) whether any candidate packagesstill need to be confirmed, and continues as described previously.

[0216] If the algorithm (1300) determines (1312) that the candidate isnot confirmed, the candidate is converted (1320) to a non-select asdescribed previously and the algorithm (1300) thereafter determines(1316) whether any candidate packages still need to be confirmed, andcontinues as described previously.

[0217] Referring to FIG. 14 shows a flowchart of an exemplary algorithm(1400) of the steps traversed in operation of competition winnertracking and payment and tracking in FIG. 3.

[0218] While the algorithm (1400) is optimized for the administration ofacademic scholarships, it is noted that with minor adaptations, thisalgorithm can monitor the continued performance of military or civilofficers, employee performance, and so on.

[0219] Competition tracking refers to the monitoring of scholarprogress, updating of competition winner files to add academicperformance and other information, and re-qualification that thecompetition winner meets the requirements of the competition system. Assuch, the competition winner tracking system must periodically updatethe competition winner files. To do this, the algorithm (1400), afterdetermining that it is time to re-qualify one or more competitionwinners (such as just before an academic year in an academic competitionsystem), sends (1402) correspondence requesting any needed information.In one embodiment relating to academic scholarships, academictranscripts are requested once a year at the completion of the year ofstudy. Further to this embodiment, prior to each further scholarshippayment, the scholar is asked whether he or she wishes to continue inthe scholarship.

[0220] In one embodiment, the system maintains a scholar tracking actionlist for scholars in order to keep track of when re-qualification isnecessary. The system determines, by accessing the scholar trackingaction list, when an action, such as requesting re-confirmationdocuments is required. In another embodiment, this information is keptseparately for each scholar.

[0221] All re-qualification documents received in support of the scholarare stored (1404) into the competition-winner segment (116) andassociated with the competition winner package. As with initialconfirmation documents, documents requested in response to updating acompetition winner's package can be received electronically or byhardcopy and are incorporated into the system by the same system andprocess as described in reference to FIGS. 4 and 5, with the differencethat the documents are stored in the scholar database and associatedwith the scholar package.

[0222] Next, a first competition winner package is selected (1406).Next, the algorithm (1400) determines (1408) whether there-qualification documents are all present and if so, the algorithm(1400) allows evaluation (1410) of the re-qualification materials todetermine if the scholar re-qualifies. Both the checks if all requireddocuments are present and the evaluation of the competition winner'sre-qualification can be implemented by automation, by human reviewers,or a combination of both. In one embodiment relating to the field ofacademic scholarship competitions, both the checking if all requireddocuments are present and the evaluation of the scholar'sre-qualification are done by human reviewers. Further to thisembodiment, the algorithm (1400) allows the human reviewers to quicklysearch in scholar packages for specific documents by keyword, documenttype, document receipt date, document originator, and so forth. Further,the algorithm (1400) allows for simultaneous display of portions of, orthe complete contents of, any re-qualification materials (or indeed anyform in the scholar package) with the re-qualification rubric and/ordecision input form.

[0223] After evaluation (1410), the algorithm (1400) determines (1412)whether the competition winner has re-qualified and if so, the newamount of the scholarship award is calculated (1414). After thescholarship award amount has been determined (1414), a check is printedand sent (1416) to the competition winner's educational facility.

[0224] Next, the algorithm (1400) determines (1418) if all competitionwinners have been passed through the re-qualification process, and ifso, the re-qualification process is finished (as shown by done indicator(1420). Otherwise, the algorithm (1400) selects (1406) another scholarpackage which has not been re-qualified and continues as describedbefore.

[0225] If the algorithm (1400) determines (1408) that not all requiredre-qualification documents are present, the algorithm (1400) alerts andfacilitates (1422) competition winner reviewers to follow up with thecompetition winner. If the re-qualification materials are competed(1424), the algorithm (1400) evaluates (1410) the scholar'sre-qualification materials and continues as discussed previously. If thealgorithm (1400) determines (1424) that the re-qualification materialsis not complete, the algorithm (1400) determines (1426) whether thecompetition winner has at least one semester of sabbatical left underthe terms of the scholarship. If the competition winner does have atleast one semester of sabbatical left, the competition winner isdesignated (1428) as being on sabbatical and the competition winner'ssabbatical allotment is decremented (1428) by one (1). Thereafter, thealgorithm (1400) determines (1418) if all competition winner packageshave been reviewed and continues as previously discussed. If thecompetition winner is determined (1426) to have no sabbaticals left, thecompetition winner's scholarship is terminated (1430) and thecompetition winner's package is updated (1430) to reflect this status.After terminating (1430) the scholar's scholarship, the algorithm (1400)determines (1418) whether any scholars have not been processed throughthe re-qualification process and continues as previously described.

[0226] If the competition winner is determined (1412) to not bere-qualified, the algorithm (1400) determines (1426) whether thecompetition winner has any sabbaticals left and continues as previouslydiscussed.

[0227] Referring to FIG. 14a, shown is a flowchart of the stepstraversed in an exemplary method (1450) for the tracking of competitionwinners and non-selected applicants.

[0228] In one embodiment for the distribution of academic scholarships,the tracking and re-qualification subsystem (260) implements trackingand information accumulation on both scholars and non-selectedapplicants. The algorithm (1450) operates by the administration ofsurveys and other information retrieval activities based on the time ofyear. The algorithm (1450) first checks (1452) if it is the start of theacademic year, and it it is, the algorithm (1450) updates scholar datathrough request (1454) of the re-qualification materials as discussed inreference to FIG. 14. As discussed in reference to FIG. 14, once there-qualification materials are received, they are incorporated (1456)into the system. If it is not (1452) the start of the academic year, thealgorithm (1450) checks (1458) if it is the time for the orientation ofnew scholars and if it is, new scholar surveys are administered (1460)to the new scholars. Thereafter, this information is incorporated (1456)into the system. If it is not (1458) orientation time, the algorithm(1450) checks (1462) if it is time for the bi-annual surveys and if itis, career development surveys (1464) and academic support surveys(1466) are administered to the active scholars. If it is not (1462) timefor the bi-annual active scholar surveys, the algorithm (1450) checks(1468) if it is graduation time for any active scholars and if it is,graduate surveys are administered (1470) to the graduating scholars.Thereafter, this information is incorporated (1456) into the system. Ifit is not (1470) a graduation time, the algorithm (1450) waits againuntil the first of the academic year (1452) and continues as previouslydescribed.

[0229] Additionally, information supplied by active scholars and scholaralumni may be received (1472) at any time. This can occur by mailed inmaterial submissions, facsimile transmissions, website submissions, andso forth. As with the previously discussed survey information, thisinformation is incorporated (1456) into the system.

[0230] Information is also received (1474) from third parties regardingnon-selected applicants. This received information is very important forverifying and improving the selection rubric and criteria employed bythe system. Non-selected applicant data is extracted from third partieswhich get this information during transactions with the non-selectedapplicants during situations such as, but not limited to, applicationsto other scholarships, credit cards, colleges or universities, etc. aswell as from targeted promotions designed to elicit trackinginformation.

[0231] Improvement of the system rubric and evaluation criteria can beeffectuated by tracking the performance of both competition winners andnon-selected applicants and comparing the later performance of bothtypes of individuals with their predicted performance as derived fromthe system rubric through the evaluation variables. To do this, eachcompetition winner's tracked performance is evaluated and scored overone or more performance variables. In one embodiment, these variablesinclude annual salary, gaps between employment periods, number of peoplemanaged, number of positive appearances on the news or on television,job satisfaction, community leadership, positive contributions tosociety, and so forth. Analysis is conducted of these performancevariable scores in reference to the individual's original evaluationvariable scores to determine the accuracy of each evaluation variable inpredicting the eventual resultant performance of the individual. Thisanalysis is conducted for all competition winners and non-selectedapplicants. The accuracy of predicting eventual resultant performance ofeach performance variable can thus be determined for each evaluationvariable. Additionally, this analysis can be conducted separately forany subgroup of competition winners and for a corresponding subgroup ofnon-selected applicants to determine the significance of each evaluationvariable in correctly differentiating individuals who went on to performsignificantly from those who did not.

[0232] Referring to FIG. 15, shown is an exemplary display presented toan evaluator. An evaluator is given the opportunity to either get anapplicant for evaluation or to quit.

[0233] Referring to FIGS. 16-17, shown are two views of an exemplarydisplay for use during determination of evaluator eligibility. Shown areselected information from an applicant package. In this example, shownare the candidate's gender, birthdate, dependency status,race/ethnicity, languages spoken, summary of high school attendance,general equivalency degree (GED) status, and future goals. Additionallyshown is evaluator eligibility question guidelines and three evaluatoreligibility questions for the evaluator to answer. Note that in otherembodiments, different questions and different numbers of totalquestions are used. Determination of evaluator eligibility andsubsequent redirection of applicant packages to another evaluator whenthe evaluator is determined ineligible to evaluate the applicant packagehelps ensure an unbiased evaluation. Additionally, evaluator eligibilityis documented in the system for later use, if needed, to counter anyreceived allegations of impropriety.

[0234] Referring to FIGS. 18-19, shown are two views of an exemplarydisplay for use during determination of candidate eligibility. Shown isinformation extracted from a candidate package and four questionsregarding eligibility of the candidate package to be evaluated. In thisexample, the questions regarding the candidate package eligibility arewhether the candidate is eligible based on high school graduation or GEDcompletion, high school grade point average (GPA), not having attendedcollege or university, and sufficient information for evaluation. Notethat in other embodiments, different questions and different numbers oftotal questions are used.

[0235] Referring to FIGS. 20-21, shown are exemplary displays for useduring evaluation of candidate packages. Shown in FIG. 20 is the typicalsplit-screen display used in one embodiment. Shown are two windows orareas. One area (the data area) contains information from thenominator's form for the candidate. The other area (the control andresponse area) contains various controls or hyperlinks which control theinformation displayed in the first area. In the shown embodiment, thecontrols allow the evaluator to control the data area to show applicantform data, nominator form data, or recommender form data. By selecting,such as by use of a mouse, an evaluator directs the system to displayinformation from the selected record in the data area. The control andresponse area also contains 11 evaluation variables requiring evaluatorresponses. In this embodiment, shown are the variables 1) curriculumrigor, 2) overall academic achievement, 3) positive self-concept, 4)realistic self-appraisal, 5) understanding/navigation of a socialsystem, 6) preference of long-term goals over short-term goals, 7)leadership experience, community service, 9) self-directed acquisitionof knowledge in a non-school field, 8) community service, 10)availability of a strong support person, 11) structure and use oflanguage in essays. By selecting a particular one of the evaluationvariables, the evaluator directs the data area to scroll to or extractinformation from the displayed record which is relevant to the selectedvariable. For example, as shown the data area of FIG. 20 showsinformation relevant to curriculum rigor. The data area shows thisinformation whenever the evaluator indicates, by selecting thecurriculum rigor variable, that the evaluator wishes to evaluate thecurriculum rigor variable.

[0236]FIG. 20a shows the screen shot of FIG. 20 with additional controlwindows. The window below and to the left of the two evaluation windowsas shown in FIG. 20 is a control window allowing the evaluator to accessthe stored image of a document from the applicant package underevaluation. This allows the evaluator to view the original documentimage which is displayed in the data area. This capability is also usedby authorized personnel when deciding situations of possible fraud. Thewindow shown to the right of the two evaluation windows as shown in FIG.20 is a control window allowing the evaluator to find a document view.As shown, a document can be found by searching such as by socialsecurity number and form type or content.

[0237] Referring to FIGS. 22-25, shown are exemplary displays ofcandidate progress detail reports. Shown in FIG. 22 is an applicantpackage progress report detailing applicant packages grouped bycomplete/incomplete and sorted by social security number (SSN). Shown inFIG. 23 is an applicant package progress report grouped bycomplete/incomplete and sorted by applicant name. Shown in FIG. 24 is anapplicant package progress report grouped by complete/incomplete, byethnicity, and sorted by social security number (SSN). Shown in FIG. 25is an applicant package progress report grouped by complete/incomplete,by ethnicity, and sorted by applicant name. In one embodiment, access tothese reports is restricted such that only system administrators areable to bring up such reports.

[0238] Referring to FIGS. 26-28, shown are exemplary displays ofcompleted applicant package reports. FIG. 26 shows a report listing allcompleted applicant packages sorted by applicant social security number.FIG. 27 shows a report listing all completed applicant packages sortedby applicant name. FIG. 28 shows a report listing all completedapplicant packages sorted by submit date of the applicant (nominee)form.

[0239] Referring to FIGS. 29-30, shown are exemplary displays ofpossible duplicate document reports. FIG. 29 shows a listing of possibleduplicate forms listed by applicant social security number. FIG. 30shows a report of possible duplicate forms listed by applicant name.

[0240] Referring to FIGS. 31-34, shown are exemplary display ofcandidate detail listing reports. Shown are candidate detail data. FIG.31 shows applicant detail data sorted by social security number. FIG. 32shows candidate detail data sorted by applicant name. FIG. 33 showsapplicant detail data ordered by one partner group sorted by socialsecurity number. FIG. 34 shows candidate detail data ordered by partnergroup and secondly by applicant name.

[0241] Referring to FIG. 35, shown is an exemplary report of a nominatorfloater with possible applicant or key form (nominee form) ordered byapplicant last name and secondly by social security number. This reportlists all nominator floaters and likely corresponding key forms (nomineeforms), if the system discovers any.

[0242] Referring to FIGS. 36-37, shown are exemplary display ofnominator floater detail reports. FIG. 36 shows a report listingnominator form floaters ordered by applicant social security number.FIG. 37 shows a report listing nominator form floaters ordered byapplicant last name and secondly by social security number.

[0243] Referring to FIGS. 38-40, shown are exemplary display ofnominator detail reports. These reports list nominator detail data. FIG.38 shows nominator detail data ordered by applicant social securitynumber. FIG. 39 shows nominator detail data ordered by applicant name.FIG. 40 shows nominator detail data ordered by nominator name.

[0244] Referring to FIG. 41, shown is an exemplary report of recommenderfloaters with possible key form (nominee form) linkages ordered byapplicant last name and secondly by social security number. This reportlists all recommender floaters and likely corresponding key forms(applicant forms), if the system discovers any.

[0245] Referring to FIGS. 42-43, shown are exemplary display ofrecommender floater detail reports. FIG. 42 shows a report listingrecommender form floaters ordered by applicant social security number.FIG. 43 shows a report listing recommender form floaters ordered byapplicant last name and secondly by social security number.

[0246] Referring to FIGS. 44-46, shown are exemplary display ofrecommender detail reports. These reports list recommender detail data.FIG. 44 shows recommender detail data ordered by applicant socialsecurity number. FIG. 45 shows recommender detail data ordered byapplicant name. FIG. 46 shows recommender detail data ordered byrecommender name.

[0247] While the invention herein disclosed has been described by meansof specific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

What is claimed is:
 1. A system for evaluating and selecting competitionwinners from a pool of candidates in a competition comprising: adatabase for storing candidate documents; a grouping subsystem coupledto said database for grouping said candidate documents into candidatedata sets; a qualification subsystem coupled to said database fordetermining said candidate data sets to be qualified or disqualified; apool reduction subsystem coupled to said database for determiningwhether said candidate data sets having been qualified meet adiscretionary eligibility standard, said discretionary eligibilitystandard varying as a function of the number of said candidate data setshaving been qualified; an evaluation subsystem coupled to saidqualification subsystem and said database for evaluating each of saidcandidate data sets having met said discretionary eligibility standardand scoring one or more evaluation variables in response to saidevaluation; and a selection subsystem coupled to said evaluationsubsystem and database for selecting one or more competition finalistsresponsive to said evaluation and scoring of said candidate data setshaving met said discretionary eligibility standard.
 2. A system as inclaim 1 further comprising: a document reception subsystem comprising: anetwork interface module coupled to said database for receivingelectronic submissions, said database storing said electronicsubmissions as candidate documents, a scanner interface module forinterfacing said system to a scanner and receiving scans of submittedhardcopy documents, and a data extraction module coupled to said scannerinterface module and said database for extracting data from said scans,said database storing said extracted data as candidate documents.
 3. Asystem as in claim 2, further comprising: a duplicate checking modulecoupled to said network interface module, said data extraction moduleand said database for determining whether said candidate documents existin said database, said candidate documents having been determined not toexist in said database being stored in said database.
 4. A system as inclaim 1 wherein said qualification subsystem comprises: a first filtermodule coupled to said database for determining whether said candidatedata sets meet a data completeness standard.
 5. A system as in claim 4wherein said data completeness standard requires said candidate datasets to include a minimum set of document types.
 6. A system as in claim4 wherein said evaluation subsystem comprises: an evaluation workstationcoupled to said database for facilitating evaluation of said candidatedata sets having met said discretionary eligibility standard, saidevaluation workstation comprising: a display for displaying informationextracted from said candidate data sets, and an input device forentering information; a matching module coupled to said database forselecting a possible evaluator to evaluate one of said candidate datasets having met said discretionary eligibility standard; an evaluatoreligibility module coupled to said database for determining whether saidevaluator is permitted to evaluate said one of said candidate data setshaving met said discretionary eligibility standard; and an evaluationmodule coupled to said evaluator eligibility module, said database, andsaid evaluation workstation for providing on said display informationextracted from said one of said candidate data sets having met saiddiscretionary eligibility standard, providing one or more evaluationvariables for scoring, and receiving a score for each of said one ormore evaluation variables from said input device.
 7. A system as inclaim 1 wherein said selection system comprises: a phase 1 selectionmodule coupled to said evaluation subsystem for selecting a first set ofcandidate finalists responsive to total scores generated responsive tocorresponding said received scores, said first set of candidatefinalists not exceeding a predetermined total number; said phase 1selection module further outputting when said first set of candidatedata sets is less than said predetermined number a sub-pool of saidcandidate data sets all having received equal corresponding said totalscores; and a phase 2 selection module coupled to said phase 1 selectionmodule and said database for selecting a second set of candidatefinalists, said second set of candidate finalists being a subset of saidsub-pool, said first set of candidate finalists plus said second set ofsaid candidate finalists equal in number to said predetermined totalnumber.
 8. A system as in claim 7, said phase 1 selection modulecomprising: a scoring module coupled to said evaluation subsystem andsaid database for generating said total scores; a grouping modulecoupled to said scoring module and said database for grouping saidcandidate data sets into score tiers responsive to corresponding saidtotal scores, each of said score tiers comprising one or more of saidcandidate data sets having equal said total scores; a selection modulecoupled to said ranking module, said ranking workstation, and saiddatabase for selecting a first set of candidate finalists responsive tosaid ranking, said first set of candidate finalists equal to or lessthan said predetermined total number; and an output module foroutputting said sub-pool, said sub-pool of candidate profiles comprisingone of said score tiers, said one of said score tiers having the highesttotal score of those of said score tiers containing candidate profilesnot selected for said first set of candidate profiles.
 9. A system as inclaim 7, said phase 2 selection module comprising: a receiving submodulecoupled to said for receiving said sub-pool of said candidate data setsfrom said phase 1 selection module; an evaluation response selectionsubmodule for determining one of said received scores to be weighted anda weight; a weighting submodule coupled to said selection module andsaid evaluation response selection submodule for weighting said one ofsaid received scores of said candidate data sets in said sub-pool; ascoring submodule coupled to said weighting submodule for generatingweighted scores for said candidate data sets in said sub-pool; agrouping submodule coupled to said scoring submodule for grouping saidcandidate data sets in said sub-pool into sub-tiers responsive tocorresponding weighted scores; a selection submodule for selectingadditional candidate finalists responsive to said weighted evaluationtier ranking; and a stop submodule for determining when a correct numberof candidate finalists have been selected and outputting said additionalcandidate finalists.
 10. A system as in claim 1 wherein said selectionsystem comprises: a scoring module for summing evaluation scores foreach of said candidate data sets having been evaluated; a score groupingmodule for grouping said candidate data sets having been evaluated intoscore tiers each comprising substantially similar corresponding ones ofevaluation score sums; a phase 1 selection module for selecting all saidcandidate data sets of the score tiers beginning with the score tierhaving the highest median evaluation score sum and proceeding to scoretiers of successively lower median evaluation score sums such that thenumber of candidate documents selected is maximized without exceeding apredetermined number, said phase 1 selection module further outputtingan active group, said active group comprising the candidate data sets ofthe score tier having the highest median evaluation score sum containingno candidates having been selected; and a phase 2 selection module forselecting a subset of said active group.
 11. A system as in claim 10wherein said phase 2 selection system comprises: a weighting module forweighting said evaluation scores for each of said candidate data sets insaid active group; a summing module for summing said weighted evaluationscores for each of said candidate data sets in said active group; ascore grouping module for grouping said candidate data sets in saidactive group into phase 2 score tiers by substantially similarcorresponding said evaluation score sums; a tie-breaking selectionmodule for selecting all candidate documents within the phase 2 scoretiers beginning with the phase 2 score tier having the highest medianevaluation score sum and proceeding to phase 2 score tiers ofsuccessively lower median evaluation score sums such that the totalnumber of selected said candidate data sets is maximized withoutexceeding said predetermined number; and a designating module fordesignating the candidate data sets of the phase 2 score tier having thehighest median evaluation score sum containing no candidates having beenselected as the active group.
 12. A system as in claim 1 furthercomprising: a monitoring subsystem coupled to said evaluation subsystemfor monitoring evaluations of qualified said candidate data sets, saidmonitoring subsystem allowing interaction with an ongoing evaluation,said monitoring subsystem allowing nullification of an evaluation whensaid evaluation is determined to be inconsistent with the requirementsof said competition.
 13. A system as in claim 1 further comprising: aconfirmation subsystem coupled to said selection subsystem and saiddatabase for determining the confirmation status of said one or morecompetition finalists and determining one or more competition winnersresponsive to said confirmation status determination.
 14. A system as inclaim 13 wherein said confirmation subsystem comprises: a workstationcomprising: a display for displaying candidate packages andcorresponding confirmation documents, and an input device for enteringinformation; a confirmation document request module coupled to saiddatabase for requesting confirmation documents; a confirmation modulecoupled to said workstation and said database for providing saidcandidate packages and said corresponding confirmation documents to saidworkstation for display and for receiving confirmation responses; and aconversion module coupled to said workstation and said database formarking ones of said candidate data sets as non-selected responsive to acorresponding confirmation response indicating nonconfirmation and formarking ones of candidate data sets as competition winners responsive toa corresponding confirmation response indicating confirmation.
 15. Asystem as in claim 1 further comprising: a competition winner trackingsubsystem coupled to said database for tracking said competition winnerscomprising: a competition winner tracking module coupled to saiddatabase for determining when update documents are required for acompetition winner, an update document request module coupled to saidcompetition-winner module for coordinating update document requestcorrespondence, and an update module coupled to said database forupdating confirmation winner packages with said required updatedocuments.
 16. A system as in claim 15, wherein said competition winnertracking subsystem further comprises: a workstation comprising: adisplay for displaying information from one of said candidate data setsand corresponding confirmation documents, and an input device forentering information; and a reconfirmation module coupled to workstationand said database for displaying the candidate data sets of saidcompetition winners with corresponding said required documents andreceiving reconfirmation responses, the status of the candidate datasets of said competition winners being updated in response to saidreconfirmation responses.
 17. A system for evaluating and selectingcompetition winners from a pool of candidates in a competitioncomprising: a document reception module for receiving documentsubmissions; a database for storing candidate records; a duplicatedetection module coupled to said document reception module and saiddatabase for determining whether any of said document submissionsduplicate any of said candidate records, said duplicate detection modulestoring said submissions determined not to be duplicates of any of saidcandidate records in said database; a qualification subsystem coupled tosaid database for determining said candidate records to be qualified ordisqualified; an evaluation subsystem coupled to said qualificationsubsystem and said database for evaluating and scoring each of saidcandidate records having been qualified over one or more evaluationvariables; and a selection subsystem coupled to said evaluationsubsystem and database for selecting one or more competition finalistsresponsive to said evaluation and scoring of said candidate recordshaving been qualified.
 18. A system as in claim 17, wherein saidcandidate records include at least one applicant form, at least onenominator form, and at least one recommender form, said system furthercomprising: a grouping module for grouping said candidate records intocandidate packages, said candidate packages comprising all of saidcandidate records relating to the same candidate; and wherein saidqualification system determines those of said candidate packages that donot have a corresponding said applicant form, said nominator form, andsaid recommender form as disqualified.
 19. A system for evaluating andselecting competition winners from a pool of candidates in a competitioncomprising: a database for storing candidate records and images ofscanned documents, said images being additionally represented bycorresponding ones of said candidate records containing data derivedfrom said images; a qualification subsystem coupled to said database fordetermining said candidate records to be qualified or disqualified; adisplay; an evaluation subsystem coupled to said qualificationsubsystem, said database, and said display for evaluating and scoringeach of said candidate records having been qualified over one or moreevaluation variables, said evaluation subsystem facilitatingpresentation on said display of information from said candidate recordshaving been qualified and corresponding said images during evaluation ofsaid candidate records having been qualified; and a selection subsystemcoupled to said evaluation subsystem and database for selecting one ormore competition finalists responsive to said scoring of said candidaterecords having been qualified.
 20. A method as in 19 wherein said one ormore evaluation variables are displayed in a first display area of saiddisplay and at least a portion of one of said images and said candidaterecords is displayed in a second display area of said display.
 21. Asystem for evaluating and selecting competition winners from a pool ofcandidates in a competition comprising: a database for storing candidaterecords; a qualification subsystem coupled to said database fordetermining said candidate records to be qualified or disqualified; agrouping subsystem coupled to said database for grouping said candidaterecords into evaluation groups responsive to information contained insaid candidate records; an evaluation subsystem coupled to saidqualification subsystem and said database for evaluating and scoringeach of said candidate records having been qualified over one or moreevaluation variables; and a selection subsystem coupled to saidevaluation subsystem and database for selecting one or more competitionfinalists for each of said evaluation groups responsive to saidevaluation and scoring of said candidate records having been qualified.22. A system as in claim 21, further comprising: an eligibilitysubsystem coupled to said database for determining said competitionrecords to be eligible or ineligible responsive to a minimum eligibilitystandard and information contained in said competition records, each ofsaid evaluation groups having a corresponding said minimum eligibilitystandard.
 23. A system as in claim 22, wherein said minimum eligibilitystandard varies with the number of candidate profiles.
 24. A system forselecting one or more candidates from a plurality of candidate profilescomprising: a storage adapted to store a plurality of candidateprofiles; an eligibility unit coupled to said storage adapted todetermine said candidate profiles to be eligible or ineligible; anevaluation unit coupled to said storage adapted to facilitate evaluationof said candidate profiles by the scoring of one or more variables byone or more evaluators; a first selection unit coupled to said storageadapted to receive said variables having been scored, said firstselection unit further adapted to select a subset of said candidateprofiles responsive to said variables having been scored, said subset ofsaid candidate profiles not exceeding a predetermined value in number;and a second selection unit coupled to said first selection unit andsaid storage adapted to receive a sub-pool of said candidate profilesand adapted to output a subset of said sub-pool of candidate profilesresponsive to application of predetermined variable weights to saidvariables having been scored, said subset of said candidate profilesadded to said subset of said sub-pool of candidate profiles equalingsaid predetermined value in number.
 25. A method as in claim 24, saidfirst selection unit comprising: a scoring module coupled to saidstorage adapted to generate total scores for said candidate profilesresponsive to said variables; a grouping module coupled to said scoringmodule and said storage for grouping said candidate profiles into scoretiers responsive to corresponding said total scores, each said scoretiers comprising one or more said candidate profiles having equal saidtotal scores; a selection module coupled to said grouping module andsaid storage for selecting a first set of candidate profiles responsiveto said grouping, said first set of candidate profiles equal to or lessthan said predetermined total number, said candidate profiles in saidfirst set of candidate profiles having corresponding said total scoreshigher than said total scores of any said candidate profiles not in saidfirst set of candidate profiles; and an output module for outputtingsaid sub-pool of candidate profiles, said sub-pool of said candidateprofiles comprising one of said score tiers, said one of said scoretiers having the highest total score of those of said score tierscontaining candidate profiles not selected for said first set ofcandidate profiles.
 26. A method as in claim 24, said second selectionunit comprising: a receiving module coupled to said for receiving saidsub-pool of said candidate profiles from said phase 1 selection module;an evaluation response selection submodule for determining one of saidone or more evaluation responses to be weighted and a weight, aweighting module coupled to said selection module and said evaluationresponse selection submodule for weighting said one of said evaluationresponses of candidate data sets in said sub-pool, a scoring modulecoupled to said weighting module for generating weighted scores for saidcandidate profiles in said sub-pool; a grouping module coupled to saidscoring module for grouping said candidate profiles in said sub-poolinto sub-tiers responsive to corresponding weighted scores, a selectionmodule for selecting additional candidate profiles responsive to saidweighted evaluation tier ranking, and a stop module for determining whena correct number of candidate profiles have been selected and outputtingsaid additional candidate profiles.
 27. A method for administering acompetition system comprising: receiving one or more candidate packages,each said candidate package containing information about a candidate ina competition; for each said candidate package: determining whether saidcandidate package meets a minimum application standard, selecting saidcandidate package when said candidate package is determined to meet saidminimum application standard, selecting an evaluator, enabling saidevaluator to evaluate said selected candidate package, and receiving anevaluation of said selected candidate package from said evaluator;ranking all said evaluated candidate packages responsive tocorresponding received said evaluations; and selecting a predeterminednumber of competition finalists responsive to said calculated overallscores.
 28. The method of claim 27 wherein said step of selecting anevaluator comprises: choosing an evaluator; determining whether saidevaluator is eligible to evaluate said selected candidate package;choosing another evaluator and returning to said step of determiningwhether said evaluator is eligible to evaluate said selected candidatepackage responsive to said determination of whether said evaluator iseligible to evaluate said selected candidate package indicating saidevaluator is not eligible to evaluate said selected candidate package;and selecting said evaluator responsive to said determination indicatingsaid evaluator is eligible to evaluate said selected candidate package.29. The method as in claim 28 wherein the step of determining whethersaid evaluator is eligible to evaluate said candidate package comprises:sending an evaluator eligibility question to said evaluator; receiving aresponse to said evaluator eligibility question from said evaluator; anddetermining whether said evaluator is eligible to evaluate said queuedcandidate package responsive to said received response regarding saidevaluator eligibility question.
 30. The method as in claim 27, whereinsaid step of receiving a plurality of candidate packages, each saidcandidate package containing information about a candidate in acompetition comprises: receiving a plurality of application documentseach relating to a candidate; and grouping said application documentssuch that all said application documents relating to the same candidateare grouped into a candidate package.
 31. The method of claim 27, saidmethod further comprising: determining whether said selected candidatepackage is eligible to be evaluated; and denying said evaluator fromevaluating said selected candidate package responsive to saiddetermination indicating said selected candidate package is not eligibleto be evaluated.
 32. The method as in claim 27 wherein the step ofdetermining whether said candidate package is eligible to be evaluatedcomprises: sending a candidate package eligibility question to saidevaluator; receiving a response from said evaluator regarding saidcandidate package eligibility question; and determining whether saidcandidate package is eligible for evaluation responsive to said receivedresponse regarding said candidate package eligibility question.
 33. Amethod as in claim 27 wherein said one or more minimum applicationstandards comprises: one or more minimum response standards; one or moreeligibility requirements; and one or more qualification requirements.34. A method as in claim 27 further comprising: determining whether anyof said competition finalists are confirmed; and selecting a newcompetition finalist for each competition finalist determined not to beconfirmed and returning to said step of determining whether any of saidcompetition finalists are confirmed.
 35. A method for selecting one ormore candidates from a plurality of candidate applications in acompetition system, comprising: determining for each of said candidateapplications whether said candidate application meets minimumqualification standards; determining the number of said candidateapplications meeting said minimum qualification standards; determiningfor each said candidate application meeting said minimum qualificationstandards whether said candidate application meets a discretionaryqualification standard, said discretionary qualification standard beinga function of said determined number of said candidate applicationsmeeting said minimum qualification standards; evaluating each saidcandidate application determined to meet said minimum qualificationstandards; scoring each of said candidate applications that has beenevaluated over a set of evaluation variables; and selecting one or moreof said candidates as competition finalists responsive to said scoring.36. A method as in claim 35 wherein said step of determining for eachsaid candidate application whether said candidate application meetsminimum qualification standards comprises: i) filtering each candidate'scandidate information to determine whether said candidate's candidateinformation meets at least one of an information completeness standardand a document completeness standard; ii) filtering said candidateinformation to ensure said candidate information indicates saidcandidate meets one or more minimum qualification requirements; iii)grouping said candidate information into two or more groups responsiveto one or more predetermined pieces of information contained in saidcandidate information; and iv) filtering said candidate information toensure said candidate information indicates said candidate meets one ormore group level standards.
 37. A method as in claim 35 wherein saidstep of filtering said candidate information to ensure said candidateinformation indicates said candidate meets one or more minimumqualification requirements comprises requiring each said candidate'scandidate information includes at least a candidate application, anomination form, and a recommendation form.
 38. A method as in claim 35wherein said step of filtering said candidate information to ensure saidcandidate information indicates said candidate meets one or more grouplevel standards comprises passing said candidate when said candidateinformation meets or exceeds at least one of a minimum GPA score, afamily income requirement, and completion of a GED.
 39. A method as inclaim 35 wherein said step of grouping comprises grouping saidcandidates by one or more of the following candidate information:geographical residence, urban/rural upbringing, primary ethnicity, andcultural background.
 40. A method as in claim 35 wherein said step offiltering said candidate information to ensure said candidateinformation indicates said candidate meets one or more group levelstandards comprises passing said candidate when said candidateinformation indicates said candidate meets or exceeds a third filterset, said third filter set comprising at least one of a minimum numberof academic awards, a minimum number of public awards, a minimum numberof honors, a minimum number of leadership roles, a minimum number ofstudent excel bubbles completed, a minimum class rigor average, aminimum amount of community service, a maximum amount of personalcircumstance exceptions, and a minimum number of paid hours ofemployment.
 41. A method as in claim 35 wherein said step of evaluatingcomprises the steps of: matching each said candidate information with areader qualified to read said candidate information, a reader beingdisqualified from reading a particular candidate's candidate informationwhen any conflicts exist; and reading each said candidate information bysaid reader, said step of reading comprising scoring said candidateinformation on a set of reading variables.
 42. A method as in claim 41wherein said set of reading variables includes the followingnon-cognitive variables: positive self-concept, realisticself-appraisal, understanding/navigation of a social system, preferenceof long-term goals over short-term goals, availability of a strongsupport person, leadership experience, community service,interest/knowledge in a non-school field.
 43. A method as in claim 35wherein said step of selecting comprises the steps of: summing saidevaluation variable scores for each of said candidate applications toproduce a total reading variable score; ranking said candidateapplications by their respective total reading variable score;determining the highest score tier having ones of said candidateapplications not already selected; determining whether the sum of allsaid candidate applications already selected plus the number of saidcandidate applications at the current score tier exceeds the totalnumber of said candidate applications to be selected; when the sum ofall said candidate applications already selected plus the number of saidcandidate applications at the current score tier is less than the totalnumber of said candidate applications to be selected, selecting thecandidate applications in the current score tier and continuing back tosaid step of determining the highest score tier; when the sum of allsaid candidate applications already selected plus the number of saidcandidate applications at the current score tier equals the total numberof said candidate applications to be selected, selecting the candidateapplications at the current score tier and aborting any furtherselection; when the sum of all said candidate applications alreadyselected plus the number of said candidate applications at the currentscore tier exceeds the total number of said candidate applications to beselected, carrying out the steps of: determining the next highest scoretier having one or more said candidate applications not alreadyselected, selecting one of said set of evaluation variables; alteringthe weight of said selected evaluation variable over all unselected saidcandidate applications in the score tier determined to be the nexthighest to produce weighted sets of reading scores, summing each saidcandidate applications weighted evaluation variable scores to produce atotal evaluation variable score; grouping said candidate applications bytheir respective total weighted reading variable scores; beginning withthe highest ranking total weighted reading variable score group andcontinuing successively to the next lower total weighted readingvariable score group, selecting the candidate applications of as manytotal weighted evaluation variable score groups as possible withoutexceeding the total number of said candidate applications to beselected, and selecting a previously-unselected evaluation variable fromsaid set of evaluation variables and returning to said step of alteringthe weight of said selected evaluation variable and continuing.
 44. Amethod as in claim 35 wherein said step of confirming comprisesrequiring each said candidate's candidate information to include aspecification of citizenship or primary residency, a tribal documentwhen said candidate is American Indian, information regarding communityservice, information regarding employment, information regarding honorsand awards, and information regarding leadership experience.
 45. Amethod of selecting a predetermined number winners from a pool ofcandidates in a competition system, said candidates having associatedvariable scores for two or more variables, said method comprising:summing the variable scores for each said candidate to produce variablescore sums; grouping said candidates into groups, each of said groupsdefining candidates having a predetermined one of said variable scoresums; determining a predetermined number of winners; determining whethersaid predetermined number of winners exceeds or equals the number ofcandidates in the group having the highest one of said variable scoresums; if said predetermined number of winners exceeds or equals thenumber of candidates in the group having the highest said variable scoresums, selecting all candidates in said group having the highest one ofsaid variable score sums; determining whether said predetermined numberof winners exceeds or equals the number of candidates already selectedplus the candidates in the group having the next highest one of saidvariable score sums; if said predetermined number of winners exceeds orequals the number of candidates already selected plus the candidates inthe group having the next highest one of said variable score sums,selecting all candidates in said group having the next highest one ofsaid variable score sums, otherwise, designating all candidates in thegroup having the highest one of said variable score sums containingcandidates not selected as the active pool and continuing to the step ofinitializing the weight of all variables; determining whether the numberof selected candidates equals the predetermined number of winners;stopping when said determination of whether the number of selectedcandidates equals the predetermined number of winners indicates thenumber of selected candidates equals the predetermined number ofwinners; designating all candidates in the next highest score groupcontaining candidates not selected as the active pool; initializing theweight of all variables; selecting a variable to be weighted, saidvariable having a weight which has not been changed since saidinitialization of the weight of all variables; weighting the variablescores for all candidates in the second-tier candidate pool bymultiplying each of the variable scores of said candidates in saidsecond-tier candidate pool by a corresponding one of said variableweights; summing the weighted variable scores for each candidate in thesecond-tier candidate pool; ranking the candidates in the second-tiercandidate pool from highest weighted variable score sum to lowestweighted variable score sum; subgrouping said candidates in thesecond-tier candidate pool such that each subgroup comprises candidateshaving the same weighted variable score sum; selecting all candidates ineach subgroup beginning with the subgroup having the highest weightedvariable score and proceeding to subgroups with successively lowerweighted variable scores such that the total number of candidatesselected in said step (e) plus the total number of candidates selectedin this step is maximized without exceeding the predetermined number ofwinners; determining whether the number of selected candidates equalsthe predetermined number of winners; stopping when said determination ofwhether the number of selected candidates equals the predeterminednumber of winners indicates the number of selected candidates equals thepredetermined number of winners; designating all candidates in thehighest variable score sum candidate subgroup having candidates notselected as the active pool; and returning to said step of selecting avariable to be weighted.
 46. A method for administering a competitionsystem comprising: receiving an application containing information abouta candidate in a competition; determining whether said application meetsa minimum application standard; retaining said application responsive tosaid determination of whether said application meets said minimumapplication standard indicating said application meets said mini mumapplication standard; reading from said application a predeterminedpiece of information; determining whether said application meets adiscretionary application standard, said discretionary applicationstandard being a function of said piece of information; retaining saidapplication responsive to said determination of whether said applicationmeets said discretionary application standard indicating saidapplication meets said discretionary application standard; selecting anevaluator; determining whether said evaluator is eligible to evaluatesaid application; enabling said evaluator to evaluate said applicationresponsive to said determination indicating said evaluator is eligibleto evaluate said application; receiving an evaluation of saidapplication from said evaluator; calculating an overall score for saidapplication responsive to said evaluation; determining a ranking of saidoverall score of said application relative to the overall scores of apool of evaluated applications; and selecting said application as acompetition winner responsive to said determination of said ranking ofsaid overall score of said application relative to the overall scores ofsaid pool of evaluated applications indicating said application meets acompetition winner ranking standard.
 47. The method as in claim 46further comprising: denying said evaluator from evaluating saidcandidate package responsive to said determination indicating saidevaluator is not eligible to evaluate said selected candidate package.48. A method for administering academic admissions comprising: receivingapplications from one or more applicants; determining which of saidapplications meet minimum application standards; retaining those of saidapplications which are determined to meet said minimum applicationstandards; reading from said applications a piece of information;determining which of said applications meet discretionary applicationstandard, said discretionary application standard being a function ofsaid piece of information; retaining those of said applications whichare determined to meet said minimum application standard; receiving arequest from an evaluator to evaluate a retained application;determining whether said evaluator is eligible to evaluate said retainedapplication; denying said evaluator from evaluating said retainedapplication responsive to said determination indicating said evaluatoris not eligible to evaluate said retained application; determiningwhether said retained application is eligible for evaluation; denyingsaid evaluator from evaluating said retained application responsive tosaid determination indicating said retained application is not eligibleto be evaluated; enabling said evaluator to evaluate said retainedapplication; receiving a response from said evaluator; determining whichof said retained applications have been successfully evaluated;producing an overall score for each said retained application which wassuccessfully evaluated; and selecting a predetermined number of saidretained application finalists responsive to said overall scores.
 49. Acompetition system user interface comprising: a data region fordisplaying information for evaluation by a user; and a evaluation regioncoupled to said data region adapted to prompt a user with at least oneevaluation inquiries, each of said evaluation inquiries causing saiddisplayed information to be adjusted so that a subset of saidinformation corresponding to an evaluation inquiry is displayedresponsive to said evaluation inquiry being selected, said evaluationregion further adapted to receive an evaluation response.
 50. Acompetition system user interface as in claim 49 further comprising: acontrol region coupled to said data region for providing a user with atleast one information source controls, each of said information sourcecontrols causing said data region to display information extracted froma corresponding source responsive to said information control beingselected, said extracted information being adjusted so that a subset ofsaid extracted information corresponding to the last selected of saidevaluation inquiries.
 51. A method for presenting a user interface to auser in a competition system comprising: providing two or more sourceaccess means; receiving a response from one of said source access means;providing information extracted from a source corresponding to said oneof said source access means responsive to said received response;providing two or more evaluation prompts; receiving a responseindicating selection of one of said evaluation prompts; providing aspecific information from said source, said specific informationcorresponding to said selected evaluation prompt; and receiving anevaluation response to said selected evaluation prompt.
 52. A method ofpreventing the accumulation of duplicate documents in a memory of acompetition system comprising: a) determining the document type of areceived document; b) extracting one or more pieces of data from saidreceived document; c) determining whether a matching document exists insaid memory having said determined document type and containing said oneor more pieces of data; d) determining whether said received document ismore complete than said matching document when said second determinationindicates said matching document exists; e) documenting said matchingdocument as duplicate when said second determination indicates saidmatching document exists and said third determination indicates saidreceived document is more complete; f) storing said received document insaid memory; and g) documenting said stored received document asduplicate when said second determination indicates said matchingdocument exists and said third determination indicates said matchingdocument is more complete.
 53. A method as in claim 52 furthercomprising: h) determining whether any documents exist in said memorycontaining the same information in one or more predetermined fields assaid stored received document and containing a difference candidatesocial security number than said stored received document; and i)flagging said stored received document and any documents determined insaid step (h) as potential fraud documents when at least one documentresults from said forth determination of said step (h).
 54. A method ofreducing the number of candidates in a candidate pool of a competitionsystem, each candidate being represented in said competition system by acorresponding candidate information package, said method comprising:determining whether each candidate information package in said candidatepool meets an information completeness standard and a documentcompleteness standard; removing all candidate information packages fromsaid candidate pool which do not meet said information completenessstandard and said document completeness standard; determining whethereach said candidate information package in said candidate pool indicatesthe corresponding candidate meets one or more minimum qualificationstandards; removing all candidate information packages from saidcandidate pool which do not indicate the corresponding candidate meetssaid minimum qualification standards; grouping said candidateinformation packages in said candidate pool into one or more groupsbased on information contained in said candidate information packages;determining whether said candidate information packages in saidcandidate pool indicate the corresponding candidate meets one or morediscretionary standards; and removing all candidate information packagesfrom said candidate pool which do not indicate the correspondingcandidate meets said one or more discretionary standards.
 55. A methodas in claim 54 wherein said step of determining whether each candidateinformation package in said candidate pool meets an informationcompleteness standard and a document completeness standard comprisesdetermining whether each said candidate information package includes atleast a candidate application, a nomination form, and a recommendationform.
 56. A method as in claim 54 wherein said step of determiningwhether each said candidate information package in said candidate poolindicates the corresponding candidate meets one or more minimumqualification standards comprises passing said candidate when saidcandidate information meets or exceeds at least one of a minimum GPAscore, a family income requirement, and completion of a GED.
 57. Amethod as in claim 54 wherein said step of grouping comprises groupingsaid candidates by one or more of geographical residence, urban/ruralupbringing, primary ethnicity, and cultural background.
 58. A method asin claim 54 wherein said step of determining whether said candidateinformation packages in said candidate pool indicate the correspondingcandidate meets one or more discretionary standards comprisesdetermining whether said candidate information indicates said candidatemeets or exceeds a third filter set, said third filter set comprising atleast one of a minimum number of academic awards, a minimum number ofpublic awards, a minimum number of honors, a minimum number ofleadership roles, a minimum number of student excel bubbles completed, aminimum class rigor average, a minimum amount of community service, amaximum amount of personal circumstance exceptions, and a minimum numberof paid hours of employment.
 59. A method as in claim 54 wherein saidstep of determining whether said candidate information packages in saidcandidate pool indicate the corresponding candidate meets one or morediscretionary standards comprises determining whether said candidateinformation indicates said candidate meets or exceeds a minimum numberof academic awards, a minimum number of public awards, a minimum numberof honors, a minimum number of leadership roles, a minimum number ofstudent excel bubbles completed, a minimum class rigor average, aminimum amount of community service, a maximum amount of personalcircumstance exceptions, and a minimum number of paid hours ofemployment.
 60. A method of reducing the number of individuals within apool of candidates, each said candidate being represented by anapplication, said method comprising: for each of said applications:reading information from said application; retrieving, as a function ofsaid information, selection criteria to apply to said application;analyzing said application with said selection criteria; and determiningwhether said application meets of said selection criteria; anddiscarding said application when said determination indicates saidapplication does not meet said selection criteria.
 61. A method ofreducing the number of individuals within a pool of candidates, eachsaid candidate being represented by an application, said methodcomprising: determining from said applications a parameter; determiningselection criteria as a function of said parameter; determining for eachsaid applications whether said application meets said selectioncriteria; and disqualifying all said applications determined not to meetsaid selection criteria.
 62. A method of matching a candidate in acompetition system to an evaluator, said candidate being represented byan application, said competition system having two or more evaluators,said method comprising: extracting a first datum from an application;determining a classification for said application responsive to saiddatum; selecting, from a plurality of evaluators preclassified as ableto evaluate applications having said determined classification, oneevaluator; providing said one evaluator with a second datum from saidapplication; receiving a response from said one evaluator indicatingwhether said one evaluator is able to evaluate said application;enabling said evaluator to evaluate said application responsive to saidreceived response.
 63. A method of training evaluators for a competitionsystem comprising: determining a number of evaluators needed; receivinga plurality of evaluator applications from evaluator candidates;evaluation of the qualification of said evaluator candidates; selectionof a set of evaluator trainees; instructing said set of evaluatortrainees with the overall nature and goals of said competition system;instructing said set of evaluator trainees with an evaluation rubric;administering paper-based evaluation exercises to said set of evaluatortrainees; monitoring and providing constructive advice during said firstadministration; administering electronic-based evaluation exercises tosaid set of evaluator trainees, said second administration substantiallymimicking an actual evaluation situation; subjecting said set ofevaluator trainees to an evaluation; selecting those said evaluatortrainees passing said evaluation.
 64. A method of evaluating candidatesfrom a pool of candidates comprising: selecting a candidate package toevaluate; matching said candidate package to an evaluator who has notpreviously evaluated or been determined ineligible to evaluate saidcandidate package; receiving an access request from said evaluator toevaluate said candidate package; providing a first set of informationfrom said candidate package and a set of evaluator eligibility questionsto said evaluator based on said candidate package; receiving a responseto said evaluator eligibility questions; determining from said responseto said evaluator eligibility questions whether said evaluator iseligible to evaluate said candidate package; denying said evaluator fromevaluating said candidate package when the result of said firstdetermination is that said evaluator is ineligible to evaluate saidcandidate package; presenting a second set of information from saidcandidate package and a set of candidate eligibility questions to saidevaluator; receiving a response to said candidate eligibility questionsfrom said evaluator; determining from said response to said candidateeligibility questions whether said candidate package is eligible forevaluation; denying said candidate package from being evaluated whensaid second determination indicates said candidate package is ineligiblefrom being evaluated; presenting a second set of information from saidcandidate package, a set of evaluation questions, and a rubric to saidevaluator; and receiving responses to said evaluation questions.
 65. Amethod for evaluation of candidates in a competition system, each ofsaid candidates being represented by a candidate package, said methodcomprising: receiving an evaluator eligibility question; answering saidevaluator eligibility question; receiving an evaluation query and afirst information from a candidate package relating to said evaluationquery, said first information including one of text and an image;receiving a data control; selecting said data control; receiving asecond information from said candidate package relating to saidevaluation query, said second information including one of said text andsaid image not provided in said first information; and providing aresponse to said evaluation query.
 66. A method as in claim 65, saidmethod further including the steps of: receiving a candidate eligibilityquestion; and responding to said candidate eligibility question.
 67. Amethod of monitoring the evaluation of individuals in a pool ofcandidates in a competition system comprising: receiving data from anevaluation; determining from said data an evaluation index; comparingsaid evaluation index to corresponding reference indicies; anddetermining whether said evaluation is aberrant responsive to saidcomparison.
 68. A method as in claim 65, wherein said evaluation indexcomprises one or more evaluation responses and said correspondingreference indicies comprise one or more of: the average correspondingevaluation responses of said evaluator for the corresponding evaluationresponses over two or more prior completed evaluations and the averagecorresponding evaluation responses for one or more other evaluators insaid competition system over two or more prior completed evaluations.69. A method as in claim 67, wherein said evaluation index comprises thenumber of reference sources checked per evaluation response and saidcorresponding reference indicies comprise a minimum number of sources tobe checked per evaluation response.
 70. A method as in claim 67, whereinsaid evaluation index comprises the average time between responses insaid evaluation and said corresponding reference indicies comprise oneor more of: the average corresponding average time between responses ofsaid evaluator over two or more prior completed evaluations and theaverage corresponding average time between responses for one or moreother evaluators in said competition system over two or more priorcompleted evaluations.
 71. A system for monitoring the evaluation ofindividuals in a pool of candidates in a competition system comprising:a display; an interface device coupled allowing a user to interact withsaid display, said display and said interface unit; receive means forelectronically receiving evaluation responses, said evaluation responsesbeing produced by an evaluator in evaluating a candidate package;processing means to produce one or more evaluation indicies from saidreceived evaluation responses; and communication means for pausing saidevaluation and interacting with said evaluator.
 72. A method of ensuringfair and consistent evaluation comprising: providing an evaluationworkstation for use by an evaluator; making available candidateapplications from one or more candidates needing evaluation; matchingsaid evaluator to one or more of said candidate applications; receivinga request from said evaluator to evaluate one of said one or morecandidate applications; providing one or more eligibility questions tosaid evaluator; receiving responses to said eligibility questions;determining from said responses to said eligibility questions whethersaid evaluator is eligible to evaluate said one candidate application;preventing said evaluator from evaluating said candidate applicationresponsive to said first determination indicating said evaluator is noteligible to evaluate said candidate application; providing one or morecandidate eligibility questions to said evaluator; receiving responsesto said candidate eligibility questions; determining from said responsesto said candidate eligibility questions whether said candidate iseligible to be evaluated; preventing said evaluator from evaluating saidcandidate application responsive to said second determination indicatingsaid candidate is not eligible to be evaluated; presenting in asubstantially simultaneous manner, said candidate application, anevaluation rubric, and a scoring display to said evaluator; monitoringthe evaluation of said candidate package by said evaluator; andinteracting with said evaluation responsive to the determination thatsaid evaluation is unfair or inconsistent.
 73. A method as in claim 72wherein said step of monitoring includes at least one of detectingwhether the average time between question responses is less than aminimum time period, determination whether the average time betweenquestion responses is less than the average time period between questionresponses for said evaluator during past evaluations, determiningwhether the average evaluation score is above a maximum score,determining whether the average evaluation score is below a minimumscore, and determination whether any evaluation score is inconsistentwith information in said candidate application.
 74. A method as in claim72 wherein said step of interacting comprises at least one ofterminating said evaluation, initiating a conversation with saidevaluator, and providing an automated message to said evaluator.
 75. Amethod of tie-breaking in a competition system containing two or morecandidates, each said candidate being represented by an application,each said application having been scored with respect to two or morereading variables, said method comprising: determining a total number ofcandidates to be selected; summing each candidate's reading variablescores to produce a total reading variable score; grouping saidcandidates into score tiers by their respective total reading variablescores; determining the score tier having the highest total readingvariable score with candidates not already selected; determining whetherthe sum of all candidates already selected plus the number of candidatesin said determined score tier exceeds the total number of candidates tobe selected; when the sum of all candidates already selected plus thenumber of candidates at the current score tier is less than the totalnumber of candidates to be selected, selecting the candidates at saiddetermined score tier and continuing back to said step of determiningthe highest score tier having candidates not already selected andcontinuing; when the sum of all candidates already selected plus thenumber of candidates at the determined score tier equals the totalnumber of candidates to be selected, selecting the candidates at thedetermined score tier and exiting said method; and when the sum of allcandidates already selected plus the number of candidates in thedetermined score tier exceeds the total number of candidates to beselected, carrying out the steps of: ascertaining the highest score tierhaving candidates not already selected, selecting one of said readingvariable scores not previously selected; altering the weight of saidselected reading variable score for all unselected candidates in saidascertained highest score tier, summing all weighted reading variablescores and all unweighted reading variable scores for each candidate insaid ascertained highest score tier to produce a corresponding totalreading variable score; grouping said candidates by their respectivetotal weighted reading variable scores into score tiers; beginning withthe score tier having the highest total weighted reading variable scoreand continuing successively to the next lower score tier by totalweighted reading variable score, selecting all candidates in all scoretiers without exceeding the total number of candidates to be selected,when the sum of all candidates already selected equals the total numberof candidates to be selected, exiting said method, and otherwise,designating the score tier having the highest total weighted readingvariable score with unselected candidates and returning to said step ofselecting one of said reading variable scores not previously selectedand continuing.
 76. A method of tracking the progress of a competitionwinner of a competition comprising: determining one or more dates whenprogress documents are to be requested; requesting on the firstoccurring of said one or more dates corresponding ones of said progressdocuments; determining whether said requested progress documents havebeen received; disqualifying said competition-winner and stopping whensaid determination indicates said requested progress documents have notall been received; evaluating said requested progress documents;determining whether said competition winner is requalified responsive tosaid requested progress document evaluation; disqualifying saidcompetition winner and stopping when said re-qualification determinationindicates said competition winner is not requalified; requesting on thenext occurring of said one or more dates corresponding ones of saidprogress documents; and returning to said step of determining whethersaid requested progress documents have been received.
 77. A method ofmanaging payment of academic scholarship awards to an academicscholarship winner, said method comprising: determining one or moredates when re-qualification documents are to be requested; requesting are-qualification document on the first occurring of said one or moredates; determining whether said requested re-qualification document hasbeen received; disqualifying said competition-winner and stopping whensaid determination of whether said requested re-qualification documenthas been received indicates said requested re-qualification document hasnot been received; evaluating said requested re-qualification document;determining whether said competition winner is requalified responsive tosaid requested requalifcation document evaluation; disqualifying saidcompetition winner and stopping when said determination of whether saidcompetition winner is requalified indicates said competition winner isnot requalified; requesting another re-qualification document on thenext occurring of said one or more dates; and returning to said step ofdetermining whether said requested re-qualification document has beenreceived.
 78. A method as in claim 77, wherein said re-qualificationdocument comprises a transcript.
 79. A method as in claim 77, whereinsaid re-qualification document comprises a tuition invoice.
 80. A methodas in claim 77, said method further comprising: issuing payment to saidacademic scholarship winner when said determination of whether saidcompetition winner is requalified indicates said competition winner isnot requalified.
 81. A method for documenting the selection ofindividuals from a pool of candidates in a competition system to combatallegations of bias, said candidates being represented by one or moreapplication documents, said method comprising: determining, prior toreception of any application documents, two or more categories to groupindividuals to promote fairness of evaluation, a minimum applicationcompleteness standard and a minimum application eligibility standard, aminimum set of evaluation variables, a selection process, and a priorityordering and weighting of said set of evaluation variables. generatingone or more dated documents documenting said determinations, each saiddated document being executed by personnel authorized by saidcompetition system. receiving a plurality of application documents;scanning in any application documents submitted in hardcopy form;storing all scanned application documents in an image file; applyingsaid minimum application completeness standard to said receivedapplication documents; applying said minimum eligibility standard tosaid application documents which passed said application of said minimumapplication completeness standard; evaluating said application documentswhich passed said application of said minimum eligibility standard; andselecting one or more individuals from said evaluated applicationdocuments using said selection process.
 82. A method of simultaneouslypromoting two or more localized sets of selection criteria duringselection of individuals from a pool of candidates in a competitionsystem, each of said candidates being represented by an application,said method comprising: grouping said individuals into two or moregroups responsive to one or more pieces of data in said applications;evaluating said applications with respect to a set of criteria; scoringsaid applications with respect to two or more global variables; applyinga weight to one of said global variables, at least one of said weightand which of said two or more global variables to have said weightapplied to being predetermined independently for each group; calculatingan overall scores for said individuals using said scored globalvariables, said calculations using the weighted value of any variableswhen available; and selecting individuals responsive to said calculatedoverall scores.
 83. A method for comparing the accuracy of predictedsuccess estimates for individuals using cognitive variables andnon-cognitive variables, said method comprising: collecting cognitivedata and non-cognitive data for a plurality of individuals; calculatinga first predicted success indicator for each of said plurality ofindividuals using corresponding said cognitive data; calculating asecond predicted success indicator for each of said plurality ofindividuals using corresponding said non-cognitive data; collectingperformance data for said plurality of individuals; analyzing saidperformance data for each of said plurality of individuals to produce ameasure of actual success; and comparing said measure of actual successwith said first predicted success indicator and said second predictedsuccess indicator to determine which of said variables produced a moreaccurate predictor of future performance.
 84. A method for comparingaccuracy of predicted success of individuals using cognitive variablesand non-cognitive variables, said method comprising: collectingcognitive data and non-cognitive data for a plurality of individuals;calculating a first predicted success indicator for each of saidplurality of individuals using corresponding said cognitive data;ranking said plurality of individuals using said first predicted successindicators; calculating a second predicted success indicator for each ofsaid plurality of individuals using corresponding said non-cognitivedata; ranking said plurality of individuals using said second predictedsuccess indicators; collecting performance data for said plurality ofindividuals; analyzing said performance data for each of said pluralityof individuals to produce a measure of actual success; ranking saidplurality of individuals using said measures of actual performance;comparing said ranking using said measures of actual success with saidranking using said first predicted success indicator and said rankingusing said second predicted success indicator; and determining therelative accuracies of said cognitive variables and said non-cognitivevariables in predicting actual performance from said comparison.
 85. Amethod as in one of claims 83-84, wherein said non-cognitive variablesare determined through the submission, for each individual, of a formfilled out by the individual, an academic recommendation form filled outby someone associated with an educational institution at which theindividual has attended, and a recommendation form filled out by someonewith personal and community knowledge of the individual.
 86. A method asin one of claims 83-84, wherein said non-cognitive variables includepositive self concept, realistic self appraisal, understanding of socialand organizational systems, ability to complete long-term goals overshort-term activities, leadership experience, community service,self-motivated acquisition of knowledge or skills, availability ofsupport person.
 87. A method as in one of claims 83-84, wherein saidactual performance data includes a college GPA.
 88. A method as in oneof claims 83-84, wherein said actual performance data includes a jobsalary.
 89. A method as in one of claims 83-84, wherein said actualperformance data includes at lease one of a measure of how oftenpromotions are received and a measure of the average length of time peremployer.
 90. A method as in one of claims 83-84, wherein said actualperformance data includes at least one of a measure of businessleadership and a measure of community standing.